# Application Lifecycle Operations

## Overview

Day-2 operations: start, stop, deploy code updates, rollback, manage snapshots, delete. Covers MSF-specific behaviors and guardrails not obvious from the API. For initial creation and IaC patterns, see [iac-and-deployment.md](iac-and-deployment.md).

## Version ID Discipline

Every `update-application`, `add-application-vpc-configuration`, and `add-application-cloud-watch-logging-option` requires `--current-application-version-id` (or `--conditional-token`). The version ID **increments after every change**. Always fetch it immediately before each update — a stale ID returns `ConcurrentModificationException` ("Exception thrown as a result of concurrent modifications to an application"). For better concurrency support in scripted/CI workflows, the API recommends `ConditionalToken` over `CurrentApplicationVersionId` — also fetched from `describe-application`.

```bash
VERSION=$(aws kinesisanalyticsv2 describe-application --application-name "$APP" \
  --query 'ApplicationDetail.ApplicationVersionId' --output text)
```

## Status Transitions and Polling

| From | To (terminal) | Trigger |
|------|---------------|---------|
| READY | STARTING → RUNNING | `start-application` |
| RUNNING | STOPPING → READY | `stop-application` |
| RUNNING | UPDATING → RUNNING | `update-application` while running |
| RUNNING | AUTOSCALING → RUNNING | autoscaling event |
| READY | UPDATING → READY | `update-application` while stopped |
| any | FORCE_STOPPING → READY | `stop-application --force` |
| RUNNING / UPDATING / AUTOSCALING | ROLLING_BACK → RUNNING | `rollback-application`, or system auto-rollback on a failed update / scaling / version upgrade |
| ROLLING_BACK | → READY or ROLLED_BACK | rollback itself failed (app moves to READY for manual remediation), or rollback completed against an app that was not running (terminal `ROLLED_BACK`) |
| READY | DELETING → *(gone)* | `delete-application` — app is removed; `describe-application` returns `ResourceNotFoundException` |
| any | MAINTENANCE → previous status | service maintenance window (transient, no action required) |

Most CLI calls return immediately. After any mutation, **poll until terminal state** before issuing the next command. The exact terminal state depends on the operation — `delete-application` has no terminal `ApplicationStatus` because the app is gone, so detect the `ResourceNotFoundException` instead of breaking on a status:

```bash
# Generic poll for start/stop/update/rollback (terminal = READY or RUNNING)
while true; do
  STATUS=$(aws kinesisanalyticsv2 describe-application --application-name "$APP" \
    --query 'ApplicationDetail.ApplicationStatus' --output text)
  case "$STATUS" in
    READY|RUNNING|ROLLED_BACK) break ;;
    *) sleep 10 ;;
  esac
done

# Poll for delete-application (terminal = app no longer exists)
while aws kinesisanalyticsv2 describe-application --application-name "$APP" \
        --query 'ApplicationDetail.ApplicationStatus' --output text 2>/dev/null; do
  sleep 10
done
```

If a transition has not completed after 10 minutes, the app is stuck — diagnose via [first-fault-isolation.md](first-fault-isolation.md) rather than retrying.

## Stop

`stop-application` without `--force` only succeeds from RUNNING. With `--force`, it stops from any state but skips a graceful savepoint, so any unflushed state since the last checkpoint is lost. Use `--force` only when the app is wedged in a transitional state.

```bash
aws kinesisanalyticsv2 stop-application --application-name "$APP" --force
```

## Start with Restore Type

`ApplicationRestoreType` controls what state the application starts from:

| Restore Type | Behavior | When to Use |
|-------------|----------|-------------|
| `RESTORE_FROM_LATEST_SNAPSHOT` (default) | Most recent successful snapshot | Normal restart |
| `RESTORE_FROM_CUSTOM_SNAPSHOT` | Specific named snapshot | Rollback to known-good state |
| `SKIP_RESTORE_FROM_SNAPSHOT` | No state — start fresh | Schema change, recovery blocker, intentional reprocess (⚠️ data loss / reprocessing) |

`AllowNonRestoredState=true` is required when the operator topology has changed (added/removed/renamed operators with `uid()`). Without it, restore fails with state-incompatibility errors.

```bash
aws kinesisanalyticsv2 start-application --application-name "$APP" \
  --run-configuration '{
    "FlinkRunConfiguration": {"AllowNonRestoredState": true},
    "ApplicationRestoreConfiguration": {"ApplicationRestoreType": "RESTORE_FROM_LATEST_SNAPSHOT"}
  }'
```

## Deploy New Code

MSF does not pull new code from S3 automatically. After uploading the new artifact, call `update-application` to point the app at the new S3 key, then **restart** to pick it up. Without a restart the app keeps running the old code.

A code update from a RUNNING state triggers a restart automatically (UPDATING → RUNNING with 10–30s downtime, varies with state size). A code update from READY does not — start the app afterward.

**Always state both behaviors when answering "how do I deploy new code":** the user's app may be RUNNING today, but the next deploy might be from READY (after a stop, or for a fresh deploy). The `update-application` → auto-restart behavior is conditional on the source state, not universal:

| Starting state | `update-application` triggers restart? | Required follow-up |
|---|---|---|
| RUNNING | Yes (UPDATING → RUNNING) | None — verify new code is live |
| READY (stopped) | No (UPDATING → READY) | Call `start-application` to pick up the new code |

```bash
aws s3 cp my-app.jar s3://$BUCKET/$KEY
aws kinesisanalyticsv2 update-application --application-name "$APP" \
  --current-application-version-id "$VERSION" \
  --application-configuration-update '{
    "ApplicationCodeConfigurationUpdate": {
      "CodeContentTypeUpdate": "ZIPFILE",
      "CodeContentUpdate": {"S3ContentLocationUpdate": {
        "BucketARNUpdate": "arn:aws:s3:::'$BUCKET'",
        "FileKeyUpdate": "'$KEY'"
      }}
    }
  }'
```

### Safe Deploy Procedure (Required Companion to Any Deploy Answer)

A safe deploy is incomplete without its rollback path. **Whenever a user asks "how do I deploy code updates safely" or any related deploy question, you MUST include both the deploy steps and the rollback options in the same answer** — they are inseparable.

**Deploy steps:**

1. (Optional but recommended) Take a snapshot from RUNNING and poll until READY. This is your fallback rollback point if both the automatic system rollback and `RollbackApplication` paths fail.
2. (One-time, recommended for production) Opt in to **automatic system rollback** by setting `ApplicationSystemRollbackConfigurationUpdate.RollbackEnabledUpdate=true` on the application. With this enabled, MSF auto-reverts failed updates, scaling actions, and version upgrades to the previous running version with minimal downtime. It is **not** on by default — existing applications must opt in.
3. Upload the new JAR to a versioned S3 key (do NOT overwrite — pointer change must be unambiguous).
4. Fetch the current `ApplicationVersionId` immediately before the update (stale ID returns `ConcurrentModificationException`).
5. Call `update-application` pointing at the new key. From RUNNING this auto-restarts; from READY call `start-application` after.
6. For state-incompatible code changes (operator topology change, removed/renamed `uid()`), set `FlinkRunConfiguration.AllowNonRestoredState=true` on restart.
7. Verify the deploy: `describe-application` shows the new `FileKey` and incremented `ApplicationVersionId`; CloudWatch shows a fresh `uptime` reset.

**Rollback options (always state these alongside the deploy steps), in priority order:**

1. **Automatic system rollback** (if opted in): MSF detects update/scaling failures (code bugs, permission issues, snapshot incompatibility on version upgrade, parallelism over `maxParallelism`, bad VPC subnets, etc.) and automatically calls `RollbackApplication` to restore the previous version with its state. If auto-rollback succeeds, the app keeps processing with minimal downtime. If auto-rollback also fails, the app transitions to READY for manual remediation. You only see this if you opted in via `ApplicationSystemRollbackConfigurationUpdate`.
2. **Manual `RollbackApplication`** (always available, no opt-in required): if the deploy succeeded but you observe downstream issues (processing errors, output regression, performance regression), call `RollbackApplication` to revert to the previous running version with its state. Monitor the operation with `DescribeApplicationOperation`. Use this when the application is stuck in a transient state, or when a deploy that completed cleanly turns out to be bad in production.
3. **Last-resort manual restore from a custom snapshot**: only if both `RollbackApplication` and the auto-rollback path failed, or if the bad code change was made many versions ago and is no longer the "previous running version." Stop with `--force`, poll until READY, `update-application` back to the previous S3 key, then start with `ApplicationRestoreType=RESTORE_FROM_CUSTOM_SNAPSHOT` pointing at the snapshot you took in step 1 above. This works only if you actually took the pre-deploy snapshot.

**Diagnose before rolling back:** the diagnostic procedure described under [Diagnosing a Failed or Unexpected Operation](#diagnosing-a-failed-or-unexpected-operation) (call `ListApplicationOperations` then `DescribeApplicationOperation` to read `statusDescription`) is the canonical first step for any failed deploy, rollback, or unexpected status transition (including `UPDATING` → `READY` when the user did not intend a no-op). Common error categories: customer code bugs (use rollback), permission issues (fix the role and retry), and MSF service issues (check AWS Health Dashboard).

## Rollback

### Diagnosing a Failed or Unexpected Operation

When a deploy goes wrong, or when an `UpdateApplication` call transitions to `UPDATING` and back without producing the expected new state, **always start by collecting diagnostic context before initiating any recovery action or assuming success**. Do not assume the operation succeeded just because the application returned to `READY` or `RUNNING` — `UpdateApplication` can transition through `UPDATING` and back even when the underlying operation failed, the new version was rolled back, or the change was rejected. Run the diagnostic flow first:

1. `ListApplicationOperations` — chronological history of all `UpdateApplication`, `Maintenance`, `RollbackApplication`, and other operations. Find the operation ID for the unexpected transition.
2. `DescribeApplicationOperation` on that operation ID — read `OperationStatus` (do not trust the application status alone) and especially `statusDescription`, which contains the actual failure reason. This is the single most informative diagnostic field MSF surfaces.
3. CloudWatch Logs for the application — read runtime errors that appear after the operation summary. Operation-level failures (IAM, parallelism limits, VPC) show up in `statusDescription`; runtime errors after a successful operation (e.g., the new code crashes on startup) show up in CloudWatch Logs.

Common failure categories surfaced this way: insufficient permissions, incompatible customer code, snapshot incompatibility on a Flink version upgrade, parallelism above `maxParallelism`, VPC misconfiguration, and MSF service issues (check AWS Health Dashboard). Each often points directly at the fix and may make rollback unnecessary.

Only after `statusDescription` is read should you decide whether to retry the operation, roll back, or fix the underlying issue. Retrying or rolling back blindly hides the root cause and tends to repeat the failure.

### Rollback Paths

MSF has three rollback paths — choose the highest one available:

1. **Automatic system rollback** (opt-in via `ApplicationSystemRollbackConfigurationUpdate.RollbackEnabledUpdate=true`). Auto-reverts **failed** `UpdateApplication`, autoscaling, or version-upgrade operations to the previous running version. Triggers when the service detects code bugs, permission issues, snapshot incompatibility on Flink version upgrade, parallelism above `maxParallelism`, or VPC misconfiguration that fails Flink job startup. **Important: this only fires when the operation itself fails.** A deploy that completes successfully but produces wrong output downstream is *not* a failed operation from MSF's perspective — auto-rollback will not engage. Use the manual `RollbackApplication` API (path 2) for that case. If auto-rollback also fails, the app moves to READY.

2. **Manual `RollbackApplication` API** (always available, no opt-in). Reverts to the previous running version with its state. Use when:
   - The deploy succeeded but the new version has downstream issues you only see in production (auto-rollback does not cover this case).
   - The application is stuck in a transient state (e.g., long UPDATING).
   - Auto-rollback was not enabled.

   ```bash
   aws kinesisanalyticsv2 rollback-application --application-name "$APP" \
     --current-application-version-id "$VERSION"
   aws kinesisanalyticsv2 describe-application-operation \
     --application-name "$APP" --operation-id "$OPERATION_ID"
   ```

3. **Manual restore from a custom snapshot** (last resort). Use only when both 1 and 2 are unavailable or have failed — for example, when the bad code change is older than the previous running version that `RollbackApplication` would target, or when both rollback paths returned errors.
   1. `stop-application --force` and poll until READY.
   2. `update-application` to point at a known-good previous S3 key.
   3. `start-application` with `ApplicationRestoreType=RESTORE_FROM_CUSTOM_SNAPSHOT` and `SnapshotName=<pre-deploy snapshot>`.

   This requires a pre-deploy snapshot. Take one before any code update so this fallback is available.

**Operation visibility for any failed deploy or rollback:** use `ListApplicationOperations` (chronological history of all `UpdateApplication`, `Maintenance`, `RollbackApplication`, and other operations) and `DescribeApplicationOperation` for the per-operation failure reason. Common error categories: customer code bugs (use rollback), permission issues (fix the role and retry), and MSF service issues (check AWS Health Dashboard).

## Runtime Properties Update

Application code reads runtime properties via `KinesisAnalyticsRuntime.getApplicationProperties()`. Update them via `EnvironmentPropertyUpdates` — the application picks up the new values on next restart. They are organized by `PropertyGroupId`, which the application code uses to look up its property map.

```bash
aws kinesisanalyticsv2 update-application --application-name "$APP" \
  --current-application-version-id "$VERSION" \
  --application-configuration-update '{
    "EnvironmentPropertyUpdates": {"PropertyGroups": [{
      "PropertyGroupId": "FlinkApplicationProperties",
      "PropertyMap": {"input.stream": "new-stream"}
    }]}
  }'
```

## Snapshots

### Snapshot vs Checkpoint

| | Checkpoint | Snapshot |
|---|---|---|
| Trigger | Automatic, periodic | Manual or stop-with-snapshot |
| Purpose | Fault tolerance | Backup, rollback, restore-on-start |
| Storage | Included in 50 GB / KPU running storage | Billed at $0.023/GB-month (durable backups) |
| Lifecycle | Managed by Flink | User must create and delete |
| Deletion | Cleared on fresh start | Deleted with the application unless preserved |

### Create

Snapshots can only be created from RUNNING. Creation is asynchronous — poll until READY.

```bash
NAME="snapshot-$(date +%Y%m%d-%H%M%S)"
aws kinesisanalyticsv2 create-application-snapshot \
  --application-name "$APP" --snapshot-name "$NAME"

# Poll until READY (timeout matters; large state can take 10+ min)
while true; do
  STATUS=$(aws kinesisanalyticsv2 list-application-snapshots --application-name "$APP" \
    --query 'SnapshotSummaries[?SnapshotName==`'"$NAME"'`].SnapshotStatus' --output text)
  [ "$STATUS" = "READY" ] && break
  [ "$STATUS" = "FAILED" ] && { echo "Snapshot failed"; exit 1; }
  sleep 5
done
```

### Delete

`delete-application-snapshot` requires the **exact** `SnapshotCreationTimestamp` from `list-application-snapshots`. Cannot delete a snapshot in CREATING state — wait for READY or FAILED first.

### Stuck CREATING

If a snapshot stays in CREATING for >10 minutes, the cause is usually:

- Backpressure slowing state serialization (check `backPressuredTimeMsPerSecond`)
- S3 permissions missing on execution role
- VPC NAT gateway down (no network path to S3)
- State too large for the snapshot timeout

Do not stop the application while a snapshot is CREATING — that risks state inconsistency. Wait or contact AWS Support.

### Retention

Snapshots are billed at $0.023/GB-month and not auto-pruned. Implement retention:

| Environment | Keep |
|-------------|------|
| Production | Last 5 + daily for 7 days |
| Staging | Last 3 |
| Development | Last 1 |

A streaming app with checkpoint-sized snapshots and no retention will accumulate cost over months. Iterate `list-application-snapshots`, filter by `SnapshotCreationTimestamp` older than threshold, delete with the exact timestamp.

## Delete Application

`delete-application` is **irreversible** and **deletes all associated snapshots** along with the application. There is no flag, grace period, or "soft delete" that preserves snapshots — they go with the app. The MSF console will warn you, but a CLI/SDK call will not.

**Before calling `delete-application`, two requirements always apply:**

1. The application **MUST be in `READY` (stopped) state.** A `RUNNING` application cannot be deleted; call `stop-application` first and poll until `ApplicationStatus=READY`.
2. The `--create-timestamp` argument **MUST exactly match** the value of `ApplicationDetail.CreateTimestamp` returned by `describe-application`. This is a guard against accidentally deleting a re-created same-named app and there is no way to bypass it.

**To preserve state across a deletion**, you must do one of these *before* calling `delete-application` — there is no way to recover snapshots after the fact:

- **(a) Create a new application from the snapshot first.** Use the existing snapshot as the basis for a new application via `CreateApplication` with the appropriate `ApplicationConfiguration` and run the new app from that snapshot. Only delete the original after the new app is verified.
- **(b) Copy the underlying S3 checkpoint/snapshot data out-of-band.** MSF stores snapshot state in S3 paths derived from your application; you can copy the relevant S3 prefixes to a bucket you control and reconstruct state later via `RESTORE_FROM_CUSTOM_SNAPSHOT` or by reading with the State Processor API. This is the fallback when option (a) isn't practical.

```bash
# 1. Confirm the app is READY (stop first if it isn't)
STATUS=$(aws kinesisanalyticsv2 describe-application --application-name "$APP" \
  --query 'ApplicationDetail.ApplicationStatus' --output text)

# 2. Pull the exact CreateTimestamp — pass this verbatim to delete-application
TIMESTAMP=$(aws kinesisanalyticsv2 describe-application --application-name "$APP" \
  --query 'ApplicationDetail.CreateTimestamp' --output text)

# 3. (Optional) Preserve state via option (a) or (b) above before deleting

# 4. Delete (irreversible — all snapshots gone)
aws kinesisanalyticsv2 delete-application --application-name "$APP" \
  --create-timestamp "$TIMESTAMP"
```

## Pre-Mutation Checklist

Before any stop, code update, scale operation, or deletion:

1. Snapshot the application (RUNNING + poll until READY)
2. Confirm no snapshots are in CREATING (deletion / stop blocks them)
3. Verify the operation is reversible — if not, confirm with the user

## Common Mistakes

| Mistake | Consequence | Prevention |
|---------|-------------|------------|
| Stale `--current-application-version-id` | `ConcurrentModificationException` | Fetch immediately before each update |
| `update-application` without restart | App keeps running old code | Restart after code update if not auto-triggered |
| Code change without `AllowNonRestoredState=true` | Restore fails on topology change | Set `true` for code updates that change operator graph |
| Stop while snapshot CREATING | State corruption risk | Block on READY status before stopping |
| Delete app to "free state" | Snapshots permanently gone | Create new app from snapshot first |
| Force stop a healthy app | Loses unflushed state since last checkpoint | Use `--force` only on stuck transitional states |
| Restore-from-snapshot after schema change | Deserialization errors | Use `SKIP_RESTORE_FROM_SNAPSHOT` and confirm reprocess with user |

## References

- [MSF API: UpdateApplication](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_UpdateApplication.html)
- [MSF API: RunConfiguration](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_RunConfiguration.html)
- [MSF Snapshots](https://docs.aws.amazon.com/managed-flink/latest/java/how-it-works-snapshots.html)

# Managed Service for Apache Flink Development Best Practices

## Overview

This guide provides Managed Service for Apache Flink-optimized development patterns, anti-patterns, and best practices for building robust, performant, and secure Flink applications on Amazon Managed Service for Apache Flink. For existing applications, use the current user's Flink version. For new applications, assume Flink 2.2 and ask if the user has a preference.

Code examples in this guide use Flink 2.2 APIs by default, which are also compatible with Flink 1.20 unless noted otherwise. See `flink-2x-migration.md` for the complete migration reference.

## Development Patterns

### Best Practices for Managed Service for Apache Flink

**Application Design**:

- Design for KPU-based automatic scaling with service-level parallelism configuration
- Use appropriate parallelism levels as suggestions (Managed Service for Apache Flink service-level settings take precedence)
- Implement proper backpressure handling for Managed Service for Apache Flink's automatic scaling algorithms
- Design stateful operations with Managed Service for Apache Flink-managed checkpoint intervals in mind

**Resource Management**:

- Configure application for KPU-based resource allocation (1 vCPU, 4GB per KPU)
- Let Managed Service for Apache Flink manage checkpoint intervals and retention through service-level configuration
- Monitor resource utilization patterns through CloudWatch metrics
- Implement proper error handling that works with Managed Service for Apache Flink's automatic recovery

**Monitoring and Alerting**:

- Leverage integrated CloudWatch dashboards and metrics
- Configure Managed Service for Apache Flink-specific alarms for KPU utilization and throughput
- Monitor key performance metrics through Managed Service for Apache Flink console and CloudWatch
- Implement application health checks that integrate with Managed Service for Apache Flink monitoring

### Managed Service for Apache Flink-Optimized Application Structure

#### Best Practice: Clean Application Architecture

```java
public class MSFStreamingApp {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        
        // NO checkpoint configuration in code - managed by MSF service
        DataStream<Event> events = env
            .fromSource(createKinesisSource(), WatermarkStrategy.forMonotonousTimestamps(), "kinesis-source")
            .uid("kinesis-source-uid");
        
        DataStream<ProcessedEvent> processed = events
            .keyBy(Event::getKey)
            .process(new EventProcessor())
            .name("event-processor")
            .uid("event-processor-uid");
        
        processed.sinkTo(createS3Sink())
            .name("s3-sink")
            .uid("s3-sink-uid");
        
        env.execute("MSF Streaming Application");
    }
}
```

`fromSource()`/`sinkTo()` are the recommended APIs for both Flink 1.20 and 2.2. The legacy `addSource()`/`addSink()` APIs are deprecated in 1.20 and removed in 2.x. See `environment-setup.md` for docker-compose.yml setup.

#### Anti-Pattern: Monolithic Processing

```java
// AVOID: Single large operator doing everything
events.map(event -> {
    // Complex transformation logic
    // Multiple business rules
    // Data enrichment
    // Validation
    // Formatting
    return processedEvent;
}); // Hard to debug, scale, and maintain
```

For state management best practices (efficient state usage, TTL, state types, Managed Service for Apache Flink state management), see [state-management.md](state-management.md).

For serialization best practices (performance hierarchy, POJO, Tuple, Avro, Protobuf, Kryo avoidance, state serialization, anti-patterns), see [serialization-guide.md](serialization-guide.md).

## Performance Best Practices

### KPU-Based Resource Configuration

#### Best Practice: Managed Service for Apache Flink KPU-Optimized Applications

```java
// Application code should be KPU-agnostic
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

// DO NOT set parallelism in application code for Managed Service for Apache Flink deployment
// Managed Service for Apache Flink manages parallelism through KPU configuration

// For local Docker development only:
if (isLocalDevelopment()) {
    int localParallelism = Math.max(1, Runtime.getRuntime().availableProcessors() - 1);
    env.setParallelism(localParallelism);
}

// Set operator-specific parallelism only when business logic or infrastructure requires it
// E.g. you may set parallelism on a Kafka source operator to be equal to number of partitions, if the overall app parallelism is higher than the number of partitions (lower parallelism for source, but enable high parallelism for processing operators with keyBy or similar operators that spread load)
dataStream
    .keyBy(Event::getPartitionKey)
    .process(new HeavyProcessor())
    // Only set if this operator specifically needs different parallelism
    .setParallelism(5);
```

### Error Handling and Recovery in Managed Service for Apache Flink

#### Best Practice: Use Side Outputs for dead letter queues for bad data handling and dependency failures

```java
public class RobustProcessor extends ProcessFunction<Event, ProcessedEvent> {
    public static final OutputTag<Event> DEAD_LETTER_TAG =
            new OutputTag<Event>("dead-letter") {};
    private transient Counter errorCounter;
    
    @Override
    public void open(OpenContext openContext) throws Exception {
        errorCounter = getRuntimeContext().getMetricGroup().counter("processing_errors");
    }
    
    @Override
    public void processElement(Event event, Context ctx, Collector<ProcessedEvent> out) {
        try {
            // Validate input
            if (!isValidEvent(event)) {
                LOG.warn("Invalid event received: {}", event);
                ctx.output(DEAD_LETTER_TAG, event);
                return;
            }
            
            ProcessedEvent result = processEvent(event);
            out.collect(result);
            
        } catch (TransientException e) {
            // Let Managed Service for Apache Flink handle transient errors through restart strategy
            LOG.warn("Transient error processing event {}, Managed Service for Apache Flink will retry", event.getId(), e);
            throw e; // Managed Service for Apache Flink restart strategy handles this
            
        } catch (Exception e) {
            // Handle permanent errors gracefully
            LOG.error("Permanent error processing event {}", event.getId(), e);
            errorCounter.inc();
            ctx.output(DEAD_LETTER_TAG, event);
            // Don't throw - continue processing other events
        }
    }
}
```

## Configuration Optimization

**Critical Principle**: Managed Service for Apache Flink applications must clearly separate local development configuration from Managed Service for Apache Flink service-level configuration. Managed Service for Apache Flink manages all advance Flink runtime parameters (i.e. `FLINK_PROPERTIES` configs such as `state.backend` and `restart-strategy`) and should not be a consideration for developers outside of local development.

- **Local Docker Configuration**: Used only for Kiro-based development with Docker containers
- **Managed Service for Apache Flink Service Configuration**: Managed through Managed Service for Apache Flink console and service APIs, not in application code
- **Application Code**: Should be environment-agnostic and avoid hardcoded infrastructure settings
- **Advanced Configs**: Configurations for the Flink runtime are managed by Managed Service for Apache Flink and should not be a consideration for developers outside of local development - some configurations can be updated via AWS Support Case requests (such as `state.backend` for RocksDB vs. HashMap) but have significant considerations to weigh for application health and stability and in general should be managed by Managed Service for Apache Flink

### Environment-Specific Configuration Management

#### Best Practice: Clean Configuration Separation

```java
import com.amazonaws.services.kinesisanalytics.runtime.KinesisAnalyticsRuntime;

public class FlinkStreamingJob {
    private static final String LOCAL_PROPS = "flink-application-properties-dev.json";
    
    private static boolean isLocal(StreamExecutionEnvironment env) {
        String runtime = System.getenv("RUNTIME_ENVIRONMENT");
        return env instanceof LocalStreamEnvironment || "local".equalsIgnoreCase(runtime);
    }

    private static Map<String, Properties> loadApplicationProperties(StreamExecutionEnvironment env) throws IOException {
        if (isLocal(env)) {
            InputStream input = FlinkStreamingJob.class.getClassLoader().getResourceAsStream(LOCAL_PROPS);
            if (input == null) throw new IOException("Unable to find " + LOCAL_PROPS);
            java.nio.file.Path tempFile = java.nio.file.Files.createTempFile("flink-app-props", ".json");
            java.nio.file.Files.copy(input, tempFile, java.nio.file.StandardCopyOption.REPLACE_EXISTING);
            input.close();
            tempFile.toFile().deleteOnExit();
            return KinesisAnalyticsRuntime.getApplicationProperties(tempFile.toString());
        } else {
            return KinesisAnalyticsRuntime.getApplicationProperties();
        }
    }

    public static void main(String[] args) throws Exception {
        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        if (isLocal(env)) {
            env.enableCheckpointing(10_000);
            env.setParallelism(3);
        }
        final Map<String, Properties> applicationProperties = loadApplicationProperties(env);
    }
}
```

In Managed Service for Apache Flink, application properties are configured at the Application level with property groups (same structure as the local JSON file).

## Anti-Patterns to Avoid

### Deployment Anti-Patterns

1. **Single-Stack IaC Without Pre-Uploaded JAR**
   The MSF application resource validates that the JAR exists in S3 at creation time. A single CloudFormation stack (or equivalent) that creates the S3 bucket and the MSF application together will fail because the JAR hasn't been uploaded yet. Always use a two-phase deployment: deploy infrastructure first, upload the JAR, then deploy the application resource. See [IaC and Deployment Guide](iac-and-deployment.md) for patterns.

### Configuration Anti-Patterns

1. **Hardcoded Infrastructure Configuration in Application Code**
2. **App-Level Savepoint Management**

### Performance Anti-Patterns

1. **KPU-Unaware Parallelism Configuration**

   ```java
   // AVOID: Fixed parallelism that doesn't align with KPU model
   env.setParallelism(7); // Doesn't align with KPU scaling
   dataStream.setParallelism(13); // Arbitrary parallelism, only set when operator requires custom parallelism
   ```

2. **Excessive Rebalancing**

   ```java
   // AVOID: Unnecessary rebalance operations
   stream.rebalance().map(...).rebalance().filter(...);
   // Breaks Managed Service for Apache Flink's automatic load balancing
   ```

3. **Blocking Operations in Processing Functions**

   ```java
   // AVOID: Synchronous external calls that block KPU resources, use Async functions instead
   public void processElement(Event event, Context ctx, Collector<Result> out) {
       Result result = externalService.blockingCall(event); // Blocks KPU
       out.collect(result);
   }
   ```

4. **Large State Objects Without TTL**

   ```java
   // AVOID: Unbounded state growth
   private transient ListState<Event> allEvents; // Can exhaust KPU memory
   private transient MapState<String, LargeObject> cache; // No TTL configured
   ```

# Flink CDC Connector Guide

This guide covers Change Data Capture (CDC) connector configuration for Apache Flink applications on Amazon Managed Service for Apache Flink. Flink CDC enables reading snapshot and incremental change data from databases without requiring Kafka or Kafka Connect — Debezium runs embedded within the Flink application.

## Overview

Flink CDC is a streaming data integration tool built on Apache Flink that captures database changes in real time. It supports two usage modes:

1. **Source Connectors** (DataStream API / Table API / SQL): Individual CDC source connectors for reading changes from a single database table or set of tables into a Flink job for custom processing. **This is the supported approach for MSF.**
2. **Pipeline API** (YAML-based, Flink CDC 3.x): End-to-end data integration pipelines defined in YAML for whole-database synchronization with schema evolution, routing, and transforms. **This does NOT run on MSF** — it requires the `flink-cdc.sh` CLI which is only available on self-managed Flink clusters.

For MSF deployments, use the Source Connector approach via DataStream API or Table API/SQL.

## Version Compatibility

**CRITICAL:** Flink CDC versions must match your Flink version. Use this mapping for MSF-supported Flink versions:

| Flink CDC Release | Flink 1.20 coordinate | Flink 2.2 coordinate | Notes |
|---|---|---|---|
| `3.6.x` | `3.6.0-1.20` | `3.6.0-2.2` | Recommended for new projects. Per-Flink-version artifacts. |
| `3.5.x` | `3.5.0` | ❌ | Flink 1.20 only. Single unsuffixed artifact. |
| `3.4.x` | `3.4.0` | ❌ | Flink 1.20 only. Single unsuffixed artifact. |
| `3.3.x` | `3.3.0` | ❌ | Flink 1.20 only (also supports 1.18, 1.19). Single unsuffixed artifact. |

For Flink 2.2 on MSF, you must use Flink CDC 3.6.x.

**IMPORTANT — version coordinate change in 3.6.x:** Starting with the 3.6.x line, Flink CDC publishes **per-Flink-version artifacts** on Maven Central. The plain `3.6.0` GAV does NOT exist — only `3.6.0-1.20` and `3.6.0-2.2`. Earlier versions (3.5.x and below) used a single artifact compatible with multiple Flink minors. Always copy the coordinate from the table above; do not assume an unsuffixed `3.6.0` will resolve.

## Supported Database Sources

| Connector | Databases | Key Mechanism |
|---|---|---|
| `mysql-cdc` | MySQL 5.6–8.0.x, Aurora MySQL, RDS MySQL, MariaDB 10.x | Binlog |
| `postgres-cdc` | PostgreSQL 9.6–14, Aurora PostgreSQL, RDS PostgreSQL | WAL / Logical Replication |
| `oracle-cdc` | Oracle 11, 12, 19, 21 | LogMiner or XStream |
| `sqlserver-cdc` | SQL Server 2012–2019 | CT (Change Tracking) |
| `mongodb-cdc` | MongoDB 3.6+ (replica set or sharded) | Change Streams |
| `db2-cdc` | Db2 11.5 | ASN Capture |

All connectors except MongoDB use Debezium under the hood.

## Maven Dependencies

Add the CDC connector for your database. The artifact version corresponds to the Flink CDC release, not the Flink version:

```xml
<!-- MySQL CDC Source -->
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-mysql-cdc</artifactId>
    <version>${flink-cdc.version}</version>
</dependency>

<!-- PostgreSQL CDC Source -->
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-postgres-cdc</artifactId>
    <version>${flink-cdc.version}</version>
</dependency>

<!-- Oracle CDC Source -->
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-oracle-cdc</artifactId>
    <version>${flink-cdc.version}</version>
</dependency>

<!-- SQL Server CDC Source -->
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-sqlserver-cdc</artifactId>
    <version>${flink-cdc.version}</version>
</dependency>

<!-- MongoDB CDC Source -->
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-mongodb-cdc</artifactId>
    <version>${flink-cdc.version}</version>
</dependency>
```

Set `flink-cdc.version` to the per-Flink-version coordinate from the table above:

```xml
<!-- Flink 1.20 -->
<flink-cdc.version>3.6.0-1.20</flink-cdc.version>
<!-- or for Flink 2.2 -->
<flink-cdc.version>3.6.0-2.2</flink-cdc.version>
```

Starting with 3.6.x, Flink CDC publishes per-Flink-version artifacts; the unsuffixed `3.6.0` GAV does not exist on Maven Central. For 3.5.x and earlier (Flink 1.20 only), use the unsuffixed coordinate (e.g., `3.5.0`). See `dependency-management.md` for the full pom.xml template.

**Note:** The MySQL JDBC driver is GPL-licensed and not bundled in the CDC connector JAR. You must add it separately:

```xml
<dependency>
    <groupId>com.mysql</groupId>
    <artifactId>mysql-connector-j</artifactId>
    <version>8.0.33</version>
</dependency>
```

## Database Credentials and Secrets Management

**Secrets Manager is the only supported credential source for CDC on MSF.** Database passwords must be fetched in application code at job startup. Do not put credentials into MSF runtime properties.

The supported pattern is to keep only non-sensitive values plus a **secret ID** in MSF runtime properties, and look up the actual credentials with the AWS SDK in `main()` before constructing the source.

### Fetch from Secrets Manager in application code

Add the AWS SDK Secrets Manager dependency:

```xml
<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>secretsmanager</artifactId>
    <version>2.25.0</version>
</dependency>
```

Resolve the secret at job startup and pass the values into the builder:

```java
import software.amazon.awssdk.services.secretsmanager.SecretsManagerClient;
import software.amazon.awssdk.services.secretsmanager.model.GetSecretValueRequest;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;

private static DbCreds loadDbCreds(String secretId) throws Exception {
    try (SecretsManagerClient sm = SecretsManagerClient.create()) {
        String json = sm.getSecretValue(
            GetSecretValueRequest.builder().secretId(secretId).build()
        ).secretString();
        JsonNode node = new ObjectMapper().readTree(json);
        return new DbCreds(node.get("username").asText(), node.get("password").asText());
    }
}

// In main(), before building the source:
DbCreds creds = loadDbCreds(cdcConfig.getProperty("secret.id"));

MySqlSource<String> mySqlSource = MySqlSource.<String>builder()
    .hostname(cdcConfig.getProperty("hostname"))
    .port(Integer.parseInt(cdcConfig.getProperty("port", "3306")))
    .databaseList(cdcConfig.getProperty("database"))
    .tableList(cdcConfig.getProperty("database") + "\\." + cdcConfig.getProperty("table"))
    // Load from SecretsManager
    .username(creds.username)
    .password(creds.password)
    // ...
    .build();
```

MSF runtime properties only carry the non-sensitive values plus the secret ID:

```json
[{
  "PropertyGroupId": "cdc.mysql.config",
  "PropertyMap": {
    "hostname": "my-aurora-cluster.cluster-xxxx.us-east-1.rds.amazonaws.com",
    "port": "3306",
    "database": "ecommerce",
    "table": "orders",
    "secret.id": "cdc-db-credentials",
    "server-id": "5400-5404"
  }
}]
```

The secret itself can still be created via CloudFormation/CDK/Terraform — only the *consumption* of the resolved value has to stay out of MSF properties:

### IAM permissions for the MSF execution role

Grant only the specific secret(s) the application uses; do not use `secretsmanager:*` or `Resource: "*"`.

```yaml
- Effect: Allow
  Action:
    - secretsmanager:GetSecretValue
  Resource:
    - !Sub "arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:cdc-db-credentials-*"
```

The trailing `-*` covers the random 6-character suffix Secrets Manager appends to secret ARNs. If you encrypt the secret with a customer-managed KMS key, also grant `kms:Decrypt` on that key ARN.

### TLS / SSL to the database

Enable TLS for every CDC connection so traffic between the MSF application and the database is encrypted in transit. The default mode for CDC on MSF is **encryption without certificate verification** (`require` for Postgres, `REQUIRED` for MySQL). MSF does not give you a stable filesystem path to drop a CA bundle on — `${user.dir}` resolves to a runtime working directory that is not the JAR location, and there is no host filesystem you can pre-populate at deploy time. The conventional Postgres/MySQL "extract the bundle to a known path and point `sslrootcert`/`trustCertificateKeyStoreUrl` at it" pattern does not work: even when the file is extracted from the JAR to `/tmp` at startup, the Debezium connectors run their internal JDBC connections from a different JVM context inside the connector, so the file lookup fails (verified: `Could not open SSL root certificate file /tmp/rds-ca-...pem`). The only way to get verify-* working on MSF is to register a custom `SSLSocketFactory` per JDBC driver that loads the bundle from the classpath, which is enough additional surface that it's an opt-in for high-MITM-risk environments rather than a default.

The defense-in-depth layers that **do** apply on MSF without a CA bundle:

- **Network isolation.** MSF runs in your VPC; restrict the database security group to accept connections only from the MSF application's security group. This is the primary control against MITM: an attacker would need to be on the network path inside your VPC, not just anywhere on the internet.
- **TLS in transit.** Even without certificate verification, the connection is encrypted, which protects credentials and replication payload from passive observation.
- **Database-side enforcement.** Set `rds.force_ssl = 1` (RDS/Aurora PostgreSQL parameter group) or `require_secure_transport = ON` (Aurora MySQL) so the database refuses any non-TLS connection. This catches client-side misconfigurations that would otherwise fall back to plaintext.

**MySQL / Aurora MySQL.** Use `sslMode=REQUIRED` on Connector/J (8.0.13+) and `database.ssl.mode=required` on Debezium. This requires TLS, skips peer certificate verification.

```java
Properties jdbcProps = new Properties();
jdbcProps.setProperty("sslMode", "REQUIRED");

Properties debeziumProps = new Properties();
debeziumProps.setProperty("database.ssl.mode", "required");

MySqlSource<String> source = MySqlSource.<String>builder()
    // host/port/user/pwd/databaseList/tableList/serverId/...
    .jdbcProperties(jdbcProps)
    .debeziumProperties(debeziumProps)
    .deserializer(new JsonDebeziumDeserializationSchema())
    .build();
```

**PostgreSQL / Aurora PostgreSQL.** Use Debezium's `database.sslmode=require`. This requires TLS, skips peer certificate verification.

```java
Properties debeziumProps = new Properties();
debeziumProps.setProperty("database.sslmode", "require");
// publication.* and other Debezium props as before

PostgresIncrementalSource<String> pg = PostgresIncrementalSource.<String>builder()
    // host/port/user/pwd/database/schemaList/tableList/slotName/...
    .debeziumProperties(debeziumProps)
    .deserializer(new JsonDebeziumDeserializationSchema())
    .build();
```

#### Stronger verification (verify-ca / verify-full / VERIFY_IDENTITY)

If `require` is not sufficient and you need chain validation, the implementation path is:

1. Bundle the RDS combined CA (`global-bundle.pem`) as a classpath resource in your application JAR (e.g., `src/main/resources/rds-ca-bundle.pem`).
2. Build an `SSLContext` from that resource at job startup, using `getClass().getResourceAsStream(...)` and a `KeyStore` populated from the PEM.
3. Register a custom `SSLSocketFactory` that returns sockets from that context, and reference it by class name in the connector config — Postgres uses `database.sslfactory=<your.class.Name>`, MySQL Connector/J uses the `socketFactory` JDBC URL parameter (or a custom `TrustManager` wired into a `KeyStore` URL the driver can resolve from the classpath).
4. Set `database.sslmode=verify-full` (Postgres) or `sslMode=VERIFY_IDENTITY` (MySQL) on top of the custom factory.

This is non-trivial because each JDBC driver has its own `SSLSocketFactory`/`TrustManager` plug point and the Debezium connector instantiates JDBC connections from inside the source operator, which means the factory class has to be on the classpath of every TaskManager and resolve the bundle without filesystem assumptions. Treat this as opt-in for high-assurance environments; it is not the default for CDC on MSF.

For RDS/Aurora, also enforce TLS at the database side (`rds.force_ssl = 1` in the RDS PostgreSQL parameter group; `require_secure_transport = ON` for Aurora MySQL). With `force_ssl = 1` the database refuses any non-TLS connection, which prevents accidentally falling back to plaintext if the client config is wrong.

## Critical: Incremental Source vs Legacy Source on Flink 2.x

Each Flink CDC artifact (`flink-connector-mysql-cdc`, `flink-connector-postgres-cdc`, etc.) ships **two parallel APIs in the same JAR**. Picking the wrong one is the most common reason CDC jobs don't compile or don't run on Flink 2.x.

**Legacy `SourceFunction`-based source** — older, single-threaded, locking snapshot for MySQL.

| Database | Class | Builder return type |
|---|---|---|
| MySQL | `org.apache.flink.cdc.connectors.mysql.MySqlSource` | `.builder()` returns a `DebeziumSourceFunction<T>` |
| Postgres | `org.apache.flink.cdc.connectors.postgres.PostgreSQLSource` | `.builder()` returns a `DebeziumSourceFunction<T>` |

**Not usable on Flink 2.x.** `SourceFunction` and `env.addSource(...)` were removed in Flink 2.0. The class is still in the artifact for backward compatibility with Flink 1.x consumers, but you cannot wire its output into a Flink 2.x job. Several builder methods on the legacy classes (e.g., Postgres `publicationName(...)`) do **not** exist on the incremental builders — if you copy a snippet that calls them, it won't compile against `3.6.0-2.2`.

**Incremental Source (FLIP-27)** — lock-free parallel snapshot, chunk-level checkpointing.

| Database | Class | Builder return type |
|---|---|---|
| MySQL | `org.apache.flink.cdc.connectors.mysql.source.MySqlSource` | `.<T>builder()` returns `MySqlSourceBuilder<T>`; `build()` returns `MySqlSource<T>` |
| Postgres | `org.apache.flink.cdc.connectors.postgres.source.PostgresSourceBuilder.PostgresIncrementalSource` | `.<T>builder()` returns `PostgresSourceBuilder<T>`; `build()` returns `PostgresIncrementalSource<T>` |

**Required on Flink 2.x.** Used with `env.fromSource(...)`. Use this for all new development on any Flink version.

Two asymmetries to be aware of:

- **Class naming.** MySQL has the same class name `MySqlSource` in two packages — disambiguate by package. Postgres uses different class names (`PostgreSQLSource` legacy vs `PostgresIncrementalSource` incremental), and the incremental class is technically an inner class of `PostgresSourceBuilder`, so the entry-point spelling is unusual: `PostgresIncrementalSource.<T>builder()`.
- **Setter coverage.** The two incremental builders share most options (host/port/user/pwd/database, schema/table list, splitSize, chunkKeyColumn, splitMetaGroupSize, distributionFactor{Upper,Lower}, fetchSize, connectTimeout, connectMaxRetries, connectionPoolSize, startupOptions, debeziumProperties, deserializer, heartbeatInterval, closeIdleReaders, skipSnapshotBackfill, scanNewlyAddedTableEnabled, assignUnboundedChunkFirst, includeSchemaChanges, serverTimeZone), but **the surfaces are not identical**. MySQL has `serverId(...)`, `databaseList(...)`, `jdbcProperties(...)`, `useLegacyJsonFormat(...)`, `parseOnLineSchemaChanges(...)`. Postgres has `slotName(...)`, `decodingPluginName(...)`, `lsnCommitCheckpointsDelay(...)`, `includePartitionedTables(...)`, `includeDatabaseInTableId(...)`. Some knobs that were first-class methods on the legacy Postgres builder (e.g., `publicationName(...)`) are **only** reachable through `debeziumProperties(...)` on the incremental builder.

Always import from the connector's `.source` sub-package, never from the top-level package.

## MySQL CDC Source Configuration

### Database Prerequisites

Before using the MySQL CDC connector, the source database must be configured:

1. **Enable binlog in ROW format** (required):

   ```sql
   -- Verify binlog is enabled and in ROW format
   SHOW VARIABLES LIKE 'log_bin';        -- Must be ON
   SHOW VARIABLES LIKE 'binlog_format';  -- Must be ROW
   ```

2. **Create a dedicated CDC user** with minimal required permissions:

   ```sql
   CREATE USER 'flink_cdc'@'%' IDENTIFIED BY '<password>';
   GRANT SELECT, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'flink_cdc'@'%';
   FLUSH PRIVILEGES;
   ```

   Note: `RELOAD` permission is NOT required when incremental snapshot is enabled (the default).

3. **For Aurora MySQL / RDS MySQL**: Binlog is enabled by default. Ensure the parameter group has `binlog_format = ROW`. For Aurora, set `binlog_replication_globaldb` to `1` if using Global Database.

4. **Recommended: Enable GTID mode** for high availability failover:

   ```
   gtid_mode = on
   enforce_gtid_consistency = on
   ```

### DataStream API Pattern

```java
// CORRECT — incremental Source (FLIP-27), usable on Flink 2.x via env.fromSource(...)
import org.apache.flink.cdc.connectors.mysql.source.MySqlSource;
// WRONG — legacy SourceFunction, not usable on Flink 2.x:
// import org.apache.flink.cdc.connectors.mysql.MySqlSource;
import org.apache.flink.cdc.debezium.JsonDebeziumDeserializationSchema;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;

// Load connection config from MSF application properties
Map<String, Properties> applicationProperties = loadApplicationProperties(env);
Properties cdcConfig = applicationProperties.get("cdc.mysql.config");

MySqlSource<String> mySqlSource = MySqlSource.<String>builder()
    .hostname(cdcConfig.getProperty("hostname"))
    .port(Integer.parseInt(cdcConfig.getProperty("port", "3306")))
    .databaseList(cdcConfig.getProperty("database"))
    .tableList(cdcConfig.getProperty("database") + "\\." + cdcConfig.getProperty("table"))
    // Load from SecretsManager
    .username(creds.username)
    .password(creds.password)
    .serverId(cdcConfig.getProperty("server-id", "5400-5404"))
    .deserializer(new JsonDebeziumDeserializationSchema())
    .build();

DataStream<String> cdcStream = env
    .fromSource(mySqlSource, WatermarkStrategy.noWatermarks(), "mysql-cdc-source")
    .uid("mysql-cdc-source-uid");
```

### Key MySQL CDC Configuration Options

| Option | Required | Default | Description |
|---|---|---|---|
| `hostname` | Yes | — | MySQL server hostname or IP |
| `port` | No | 3306 | MySQL server port |
| `username` | Yes | — | MySQL user with replication permissions |
| `password` | Yes | — | MySQL user password |
| `database-name` | Yes | — | Database name (supports regex for multi-database) |
| `table-name` | Yes | — | Table name (supports regex, format: `db\.table`) |
| `server-id` | Recommended | Random 5400-6400 | Unique server ID or range (e.g., `5400-5404` for parallelism 4) |
| `scan.incremental.snapshot.enabled` | No | `true` | Enable lock-free parallel snapshot reading |
| `scan.incremental.snapshot.chunk.size` | No | 8096 | Rows per snapshot chunk |
| `scan.startup.mode` | No | `initial` | Startup mode: `initial`, `earliest-offset`, `latest-offset`, `specific-offset`, `timestamp` |
| `server-time-zone` | No | System default | MySQL server timezone (e.g., `UTC`) |
| `heartbeat.interval` | No | `30s` | Heartbeat interval for binlog position tracking |

### CRITICAL: Server ID Configuration

Every MySQL CDC reader needs a globally unique server ID across all clients connected to the MySQL cluster. When running with parallelism > 1 and incremental snapshot enabled, you must specify a server ID range:

```java
// For parallelism of 4, provide a range of at least 4 IDs
.serverId("5400-5404")
```

If multiple Flink CDC jobs read from the same MySQL instance, their server ID ranges must not overlap. Overlapping server IDs cause the error: `A slave with the same server_uuid/server_id as this slave has connected to the master`.

## PostgreSQL CDC Source Configuration

### Database Prerequisites

1. **Set WAL level to logical** (requires restart):

   ```sql
   -- Check current setting
   SHOW wal_level;  -- Must be 'logical'
   
   -- For RDS/Aurora PostgreSQL: set rds.logical_replication = 1 in parameter group
   ```

2. **Set table replica identity to FULL** (required for UPDATE/DELETE events):

   ```sql
   ALTER TABLE my_schema.my_table REPLICA IDENTITY FULL;
   ```

   Without `FULL`, Debezium cannot capture the before-image of UPDATE/DELETE events, which will cause deserialization failures.

3. **Create a dedicated CDC user**:

   ```sql
   CREATE ROLE flink_cdc WITH LOGIN PASSWORD '<password>' REPLICATION;
   GRANT SELECT ON ALL TABLES IN SCHEMA public TO flink_cdc;
   ```

4. **Ensure sufficient replication slots** (`max_replication_slots` and `max_wal_senders`):

   ```sql
   SHOW max_replication_slots;  -- Default is 10, increase if needed
   SHOW max_wal_senders;        -- Must be >= max_replication_slots
   ```

### DataStream API Pattern

Use the **incremental** `PostgresIncrementalSource` (inner class of `PostgresSourceBuilder`). The legacy `PostgreSQLSource` returns a `DebeziumSourceFunction` and is not usable on Flink 2.x. See the "Incremental Source vs Legacy Source on Flink 2.x" section above.

```java
// CORRECT — incremental Source (FLIP-27), usable on Flink 2.x via env.fromSource(...)
import org.apache.flink.cdc.connectors.postgres.source.PostgresSourceBuilder;
import org.apache.flink.cdc.connectors.postgres.source.PostgresSourceBuilder.PostgresIncrementalSource;
// WRONG — legacy SourceFunction, not usable on Flink 2.x:
// import org.apache.flink.cdc.connectors.postgres.PostgreSQLSource;
import org.apache.flink.cdc.debezium.JsonDebeziumDeserializationSchema;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;

Properties cdcConfig = applicationProperties.get("cdc.postgres.config");

// publication.name and publication.autocreate.mode are NOT builder setters on
// PostgresSourceBuilder — they are passed through debeziumProperties(...).
// (The legacy PostgreSQLSource.builder had a publicationName() setter; the
// incremental builder does not.)
Properties debeziumProps = new Properties();
debeziumProps.setProperty("publication.name", "flink_cdc_publication");
debeziumProps.setProperty("publication.autocreate.mode", "filtered");
debeziumProps.setProperty("decimal.handling.mode", "string");
debeziumProps.setProperty("time.precision.mode", "connect");

PostgresIncrementalSource<String> pgSource =
    PostgresIncrementalSource.<String>builder()
        .hostname(cdcConfig.getProperty("hostname"))
        .port(Integer.parseInt(cdcConfig.getProperty("port", "5432")))
        .database(cdcConfig.getProperty("database"))
        // Load from SecretsManager
        .username(creds.username)
        .password(creds.password)
        .schemaList(cdcConfig.getProperty("schema", "public"))
        .tableList(cdcConfig.getProperty("schema", "public") + "."
            + cdcConfig.getProperty("table"))
        .slotName(cdcConfig.getProperty("slot.name", "flink_cdc_slot"))
        .decodingPluginName("pgoutput")  // pgoutput for PostgreSQL 10+
        .deserializer(new JsonDebeziumDeserializationSchema())
        .debeziumProperties(debeziumProps)
        .includeSchemaChanges(false)
        .build();

DataStream<String> cdcStream = env
    .fromSource(pgSource, WatermarkStrategy.noWatermarks(), "postgres-cdc-source")
    .uid("postgres-cdc-source-uid");
```

### Builder methods vs Debezium properties

The two incremental builders cover most settings as first-class methods, but a few important Postgres knobs are only reachable through `debeziumProperties(Properties)`. Use this table to decide where a given option goes:

| Setting | MySQL incremental | Postgres incremental |
|---|---|---|
| Connection (host/port/user/pwd/database) | builder method | builder method |
| Tables / schemas | `databaseList(...)`, `tableList(...)` | `schemaList(...)`, `tableList(...)` |
| Server ID range | `serverId(...)` | n/a |
| Replication slot | n/a | `slotName(...)` |
| Decoding plugin | n/a | `decodingPluginName("pgoutput")` |
| Publication name | n/a | `debeziumProperties` → `publication.name` |
| Publication auto-create | n/a | `debeziumProperties` → `publication.autocreate.mode` |
| Snapshot chunk size | `splitSize(int)` | `splitSize(int)` |
| Decimal / time encoding | `debeziumProperties` → `decimal.handling.mode`, `time.precision.mode` | same |
| Startup mode | `startupOptions(...)` | `startupOptions(...)` |
| Newly added tables | `scanNewlyAddedTableEnabled(true)` | `scanNewlyAddedTableEnabled(true)` |

### CRITICAL: PostgreSQL Replication Slot Management

Replication slots are a limited resource (default max: 10) and have significant operational implications:

- **Slots retain WAL segments** until the consumer confirms processing. If a Flink job stops or falls behind, WAL accumulates on disk and can fill the volume.
- **Flink CDC does NOT automatically drop replication slots** when the job stops. You must clean up orphaned slots manually:

  ```sql
  -- List active replication slots
  SELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
  FROM pg_replication_slots;
  
  -- Drop an inactive slot
  SELECT pg_drop_replication_slot('flink_cdc_slot');
  ```

- **Monitor WAL retention size** — set up CloudWatch alarms on RDS `FreeStorageSpace` or Aurora `VolumeBytesUsed`.
- **Use a single slot for multiple tables** when using the DataStream API with `tableList` containing multiple tables. This is more efficient than one slot per table.

## Table API / SQL CDC Source

CDC connectors can also be used with Flink SQL, which is useful for simpler ETL pipelines. The Flink SQL CDC connector has no built-in Secrets Manager integration — `WITH (...)` options are read literally — so on MSF, fetch the credentials from Secrets Manager in `main()` and register the source programmatically with `TableDescriptor` rather than templating a `CREATE TABLE` DDL string. `TableDescriptor` (Flink 1.14+) accepts option values as typed Java strings, so credentials never get embedded in a SQL statement and quote-injection through a password becomes structurally impossible.

```java
import org.apache.flink.table.api.DataTypes;
import org.apache.flink.table.api.Schema;
import org.apache.flink.table.api.TableDescriptor;
import static org.apache.flink.table.api.Expressions.$;

DbCreds creds = loadDbCreds(cdcConfig.getProperty("secret.id"));  // see Database Credentials section

tableEnv.createTable("orders_cdc", TableDescriptor.forConnector("mysql-cdc")
    .schema(Schema.newBuilder()
        .column("order_id", DataTypes.INT().notNull())
        .column("customer_id", DataTypes.INT())
        .column("order_date", DataTypes.TIMESTAMP(3))
        .column("total_amount", DataTypes.DECIMAL(10, 2))
        .column("status", DataTypes.STRING())
        .columnByMetadata("db_name", DataTypes.STRING(), "database_name", true)
        .columnByMetadata("table_name", DataTypes.STRING(), "table_name", true)
        .columnByMetadata("op_ts", DataTypes.TIMESTAMP_LTZ(3), "op_ts", true)
        .primaryKey("order_id")
        .build())
    .option("hostname", cdcConfig.getProperty("hostname"))
    .option("port", "3306")
    // Load from SecretsManager — passed as typed option values, no quoting/escaping needed
    .option("username", creds.username)
    .option("password", creds.password)
    .option("database-name", "ecommerce")
    .option("table-name", "orders")
    .option("server-id", "5400-5404")
    .build());

// Subsequent SQL queries reference the registered table by name:
tableEnv.executeSql("SELECT order_id, customer_id, total_amount, status FROM orders_cdc").print();

// Or use the Table API directly:
tableEnv.from("orders_cdc")
    .select($("order_id"), $("customer_id"), $("total_amount"), $("status"))
    .execute()
    .print();
```

Why `TableDescriptor` is preferred over string-templated DDL on MSF:

- Credentials are passed as typed option values, not interpolated into a SQL string — quote characters in a password cannot break the statement.
- Schema typos fail at compile time, not at job submission.
- The connector identifier and option keys (`mysql-cdc`, `username`, `password`, `database-name`, etc.) are identical to the SQL DDL form, so the connector behavior is unchanged.
- Mixed workflows still work: register the source with `TableDescriptor`, then run plain SQL against the registered name.

### Fallback: string-templated `CREATE TABLE` DDL

If you must keep the source definition in DDL form (for parity with a `.sql` file or an existing pipeline), pass the rendered statement through `TableEnvironment.executeSql(String)` and treat the credentials with care:

```java
DbCreds creds = loadDbCreds(cdcConfig.getProperty("secret.id"));

tableEnv.executeSql(
    "CREATE TABLE orders_cdc (" +
    "  order_id INT," +
    "  customer_id INT," +
    "  order_date TIMESTAMP(3)," +
    "  total_amount DECIMAL(10, 2)," +
    "  status STRING," +
    "  db_name STRING METADATA FROM 'database_name' VIRTUAL," +
    "  table_name STRING METADATA FROM 'table_name' VIRTUAL," +
    "  op_ts TIMESTAMP_LTZ(3) METADATA FROM 'op_ts' VIRTUAL," +
    "  PRIMARY KEY (order_id) NOT ENFORCED" +
    ") WITH (" +
    "  'connector' = 'mysql-cdc'," +
    "  'hostname' = '" + cdcConfig.getProperty("hostname") + "'," +
    "  'port' = '3306'," +
    "  'username' = '" + creds.username + "'," +
    "  'password' = '" + creds.password + "'," +
    "  'database-name' = 'ecommerce'," +
    "  'table-name' = 'orders'," +
    "  'server-id' = '5400-5404'" +
    ")"
);
```

When using the DDL fallback, ensure the Secrets Manager-generated password excludes `'` (use `ExcludePunctuation` or an explicit `ExcludeCharacters` list) so a single quote cannot terminate the option value early, and never log or print the rendered DDL string. For production CDC pipelines, prefer `TableDescriptor` (or the DataStream API) so credentials never enter a string-templated statement at all.

## MSF-Specific Considerations for CDC Workloads

### VPC Configuration (REQUIRED)

CDC workloads on MSF require VPC configuration to reach the source database. The MSF application must be deployed in a VPC with network connectivity to the database:

- **RDS/Aurora in same VPC**: Configure MSF application with the same VPC and subnets. Ensure the database security group allows inbound connections from the MSF application security group on the database port (3306 for MySQL, 5432 for PostgreSQL, etc.).
- **RDS/Aurora in different VPC**: Set up VPC peering or Transit Gateway between the MSF VPC and the database VPC. Update route tables and security groups accordingly.
- **On-premises databases**: Use AWS Direct Connect or Site-to-Site VPN to establish connectivity from the MSF VPC to the on-premises network.
- **Public databases (not recommended for production)**: MSF runs in a VPC without direct internet access. You would need a NAT Gateway for outbound connectivity, but this adds latency and cost. Prefer private connectivity.

### IAM Permissions

The MSF application's IAM execution role needs permissions for:

- VPC networking (automatically managed by MSF when VPC is configured)
- AWS Secrets Manager secrets holding database credentials (always required — see [Database Credentials and Secrets Management](#database-credentials-and-secrets-management))
- S3 access for sinks (if writing CDC data to S3/Iceberg)
- KMS Decrypt on the customer-managed key encrypting the secret, if one is used

### Checkpoint Configuration

CDC workloads have specific checkpointing requirements:

- **Checkpoints are REQUIRED** for CDC to transition from snapshot phase to incremental (binlog/WAL) phase. Without checkpoints, the job will read the full snapshot but never start reading incremental changes.
- **MSF manages checkpoint intervals** at the service level. The default 60-second interval works for most CDC workloads. Do NOT call `env.enableCheckpointing(...)` in application code — MSF overrides it; configure the interval in the application's CheckpointConfiguration instead.
- **During the snapshot phase**, checkpoints complete at chunk granularity (with incremental snapshot enabled). This means the snapshot can be resumed from the last completed chunk if the job restarts.
- **Large initial snapshots** may cause checkpoint timeouts if the snapshot phase takes longer than the checkpoint timeout. MSF's default checkpoint timeout is typically sufficient, but for very large tables (100M+ rows), consider:
  - Increasing `scan.incremental.snapshot.chunk.size` to process larger chunks
  - Using `scan.startup.mode = 'latest-offset'` to skip the snapshot entirely if historical data is not needed
  - Breaking the snapshot into multiple jobs by table

#### Diagnosing "snapshot completed but binlog never starts"

This is the single most common CDC symptom on MSF. The diagnostic path is always the same:

1. Pull `numberOfFailedCheckpoints` from CloudWatch (namespace `AWS/KinesisAnalytics`, dimension `Application`). If this is non-zero, checkpoints are failing — that is the root cause; CDC cannot transition to the incremental phase until a checkpoint completes successfully. Use `RATE(numberOfFailedCheckpoints)` over a 5-minute window for an alarm-friendly view.
2. Pull `lastCheckpointDuration` and compare to the checkpoint interval. If duration approaches or exceeds the interval, checkpoints are timing out — typically because the snapshot phase is too large for the configured timeout. Increase `scan.incremental.snapshot.chunk.size`, or raise the checkpoint timeout, or both.
3. Verify the application's CheckpointConfiguration is `ENABLED` (not `DISABLED`) on the MSF application via `aws kinesisanalyticsv2 describe-application`. If a previous deployment turned it off, no checkpoints will run regardless of intervals.
4. Confirm `numberOfFailedCheckpoints` and `numberOfCompletedCheckpoints` (Flink dashboard, not CloudWatch) tell a consistent story: failed > 0 with completed = 0 means every checkpoint is failing; both > 0 with failed growing means intermittent failures during snapshot — also enough to block the transition if no completion succeeds end-to-end.

Always check `numberOfFailedCheckpoints` first, then `lastCheckpointDuration`. Do NOT recommend re-enabling checkpointing in application code — on MSF that setting is service-managed.

### KPU Sizing for CDC Workloads

CDC workloads have unique resource characteristics:

- **Source parallelism**: MySQL CDC with incremental snapshot supports parallel reading during the snapshot phase. Set source parallelism to match the number of server IDs in your range. During the binlog phase, reading is single-threaded regardless of parallelism.
- **PostgreSQL CDC**: Source parallelism is effectively 1 for the WAL reader. Downstream operators can have higher parallelism after a `keyBy`.
- **Memory considerations**: The snapshot phase buffers chunk data in memory. For tables with wide rows or large text/blob columns, allocate additional KPUs.
- **Recommended starting point**: 2-4 KPUs for a single-table CDC workload, 4-8 KPUs for multi-table workloads. Monitor backpressure and adjust.

## Common Gotchas and Troubleshooting

### Snapshot Phase Issues

#### Problem: Snapshot takes too long or causes checkpoint timeouts

- The initial snapshot reads the entire table before switching to incremental mode.
- With incremental snapshot enabled (default), the snapshot is chunked and checkpointable.
- For very large tables, increase `scan.incremental.snapshot.chunk.size` or use `scan.startup.mode = 'latest-offset'` to skip the snapshot.

#### Problem: `FLUSH TABLES WITH READ LOCK` errors

- This only occurs when `scan.incremental.snapshot.enabled = false` (the old snapshot mechanism). Keep incremental snapshot enabled (the default) to avoid global locks.

#### Problem: Snapshot restarts from the beginning after job failure

- Ensure checkpointing is enabled. Without checkpoints, snapshot progress is not persisted.
- With incremental snapshot, progress is checkpointed at chunk granularity.

### Binlog / WAL Phase Issues

#### Problem: `binlog file has been purged` or `WAL segment has been removed`

The Flink job fell too far behind and the database cleaned up the binlog/WAL files it needed to resume from. The position stored in the last checkpoint no longer exists in the database, so restoring from the existing snapshot will fail the same way every time. Recovery has three parts — apply all three:

1. **Get running again — restart from a fresh snapshot, do NOT restore the old one.** Stop the application and start it without the old snapshot, with `scan.startup.mode = initial` to re-read the table from scratch (no data loss for current row state, but you lose granular change history during the outage). If a gap is acceptable, `scan.startup.mode = latest-offset` is faster but lossy. Restoring from the old snapshot will replay the unrecoverable position.

2. **Increase binlog / WAL retention so future outages have headroom.**

   ```sql
   -- MySQL (community / self-managed): increase binlog retention (e.g., 7 days)
   SET GLOBAL expire_logs_days = 7;
   -- MySQL 8.0+:
   SET GLOBAL binlog_expire_logs_seconds = 604800;

   -- Aurora MySQL: use the Aurora-specific procedure (expire_logs_days /
   -- binlog_expire_logs_seconds are not honored on Aurora MySQL)
   CALL mysql.rds_set_configuration('binlog retention hours', 168);  -- 7 days
   CALL mysql.rds_show_configuration;

   -- PostgreSQL (RDS): increase WAL retention
   -- Set wal_keep_size in the parameter group (e.g., 2048 MB)
   ```

   Pick a window that comfortably exceeds your worst-case outage plus reprocessing time.

3. **Right-size KPUs so steady-state lag stays well within retention.** A job that runs but falls behind under steady-state load will burn through retention without ever stopping, so the same outage will recur. See the [KPU Sizing for CDC Workloads](#kpu-sizing-for-cdc-workloads) section in this guide for sizing guidance, and [scaling-decisions.md](scaling-decisions.md) for the operational scale-up workflow. Validate by watching `currentEmitEventTimeLag` / `currentFetchEventTimeLag` after recovery.

Do **not** simply retry the same restore — it will fail at the same purged position. Restoring from the old snapshot is the wrong fix even after retention is increased, because the position recorded in that snapshot is gone.

#### Problem: MySQL `server_id` conflicts

- Each CDC reader needs a unique server ID. If multiple jobs or tools connect to the same MySQL instance, their server IDs must not overlap.
- Use explicit server ID ranges: `.serverId("5400-5404")` for parallelism 4.

#### Problem: PostgreSQL WAL disk usage growing unbounded

- Replication slots retain WAL until the consumer confirms. If the Flink job is stopped or slow, WAL accumulates.
- Monitor replication slot lag and set up alerts.
- Clean up orphaned slots when jobs are permanently stopped.

### Schema Evolution

#### Problem: Source table schema changes (DDL) break the CDC job

- By default, Flink CDC source connectors do NOT automatically handle schema changes in the DataStream API. A column addition or type change in the source table can cause deserialization errors.
- Mitigation strategies:
  1. Use `JsonDebeziumDeserializationSchema` which is more tolerant of schema changes (new fields appear in JSON, removed fields disappear).
  2. For the Pipeline API (YAML), schema evolution is supported with configurable behaviors: `evolve`, `try_evolve`, `lenient`, `ignore`, or `exception`.
  3. Plan for schema changes by using a flexible deserialization approach and validating downstream compatibility before applying DDL.

### Deserialization Performance

#### Problem: High CPU usage from JSON serialization/deserialization

- `JsonDebeziumDeserializationSchema` serializes Debezium records to JSON strings, which then need to be parsed again downstream. This double serialization can consume 60%+ of CPU.
- For high-throughput workloads, consider:
  1. Implementing a custom `DebeziumDeserializationSchema` that converts directly to your target POJO type.
  2. Using `RowDataDebeziumDeserializeSchema` for Table API integration (avoids JSON intermediate format).
  3. Enabling Flink object reuse (`env.getConfig().enableObjectReuse()`) to reduce serialization overhead between operators with the same parallelism.

### Ordering Guarantees

- During the **snapshot phase**, records are read via `SELECT` queries with no guaranteed ordering. If ordering matters, use `keyBy` on the primary key downstream.
- During the **incremental phase** (binlog/WAL), records arrive in commit order from the database. A single CDC source reader preserves this order.
- After a `keyBy` or parallelism change, ordering is preserved per-key but not globally.

## Complete MSF Application Example: MySQL CDC to Processing

```java
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.cdc.connectors.mysql.source.MySqlSource;
import org.apache.flink.cdc.debezium.JsonDebeziumDeserializationSchema;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import com.amazonaws.services.kinesisanalytics.runtime.KinesisAnalyticsRuntime;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;

import java.util.Map;
import java.util.Properties;

public class MySqlCdcApplication {

    private static final ObjectMapper MAPPER = new ObjectMapper();

    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        // Load application properties (MSF runtime or local dev)
        Map<String, Properties> applicationProperties = loadApplicationProperties(env);
        Properties cdcConfig = applicationProperties.get("cdc.mysql.config");

        // Build MySQL CDC source
        MySqlSource<String> mySqlSource = MySqlSource.<String>builder()
            .hostname(cdcConfig.getProperty("hostname"))
            .port(Integer.parseInt(cdcConfig.getProperty("port", "3306")))
            .databaseList(cdcConfig.getProperty("database"))
            .tableList(cdcConfig.getProperty("database") + "\\." + cdcConfig.getProperty("table"))
            // Load from SecretsManager
            .username(creds.username)
            .password(creds.password)
            .serverId(cdcConfig.getProperty("server-id", "5400-5404"))
            .deserializer(new JsonDebeziumDeserializationSchema())
            .includeSchemaChanges(false)
            .build();

        // Create CDC stream
        env.fromSource(mySqlSource, WatermarkStrategy.noWatermarks(), "mysql-cdc-source")
            .uid("mysql-cdc-source-uid")
            .map(json -> {
                JsonNode node = MAPPER.readTree(json);
                // Extract the "after" image (current row state)
                JsonNode after = node.get("after");
                String op = node.get("op").asText();
                // op: "r" = read (snapshot), "c" = create, "u" = update, "d" = delete
                return new CdcEvent(op, after != null ? after.toString() : null);
            })
            .name("parse-cdc-events")
            .uid("parse-cdc-events-uid")
            .filter(event -> event.getAfter() != null) // Filter out deletes if not needed
            .name("filter-deletes")
            .uid("filter-deletes-uid")
            .keyBy(CdcEvent::getKey)
            .process(new CdcProcessingFunction())
            .name("process-cdc-events")
            .uid("process-cdc-events-uid")
            .sinkTo(createSink())
            .name("output-sink")
            .uid("output-sink-uid");

        env.execute("MySQL CDC Application");
    }

    // See best-practices.md for loadApplicationProperties pattern
    private static Map<String, Properties> loadApplicationProperties(
            StreamExecutionEnvironment env) throws Exception {
        // ... standard MSF property loading pattern
    }
}
```

### MSF Application Properties Configuration

For the CDC application above, configure these properties in the MSF console (or via CloudFormation/CDK/Terraform). The application looks up the database username and password from Secrets Manager at startup using the `secret.id` property — see [Database Credentials and Secrets Management](#database-credentials-and-secrets-management) for the full pattern, IAM, and rationale.

```json
[
  {
    "PropertyGroupId": "cdc.mysql.config",
    "PropertyMap": {
      "hostname": "my-aurora-cluster.cluster-xxxx.us-east-1.rds.amazonaws.com",
      "port": "3306",
      "database": "ecommerce",
      "table": "orders",
      "secret.id": "cdc-db-credentials",
      "server-id": "5400-5404"
    }
  }
]
```

For local development, create `flink-application-properties-dev.json` with the same structure pointing to a local or Docker MySQL instance.

## CDC Anti-Patterns

### Anti-Pattern: Missing Operator UIDs on CDC Sources

```java
// AVOID: No UID means state cannot be restored after code changes
env.fromSource(mySqlSource, WatermarkStrategy.noWatermarks(), "source");

// CORRECT: Always set UIDs for stateful operators
env.fromSource(mySqlSource, WatermarkStrategy.noWatermarks(), "mysql-cdc-source")
    .uid("mysql-cdc-source-uid");
```

### Anti-Pattern: Skipping Checkpoints for CDC

```java
// AVOID: No checkpointing — CDC will never transition to incremental phase
// and snapshot progress will be lost on restart

// CORRECT: Checkpointing is managed by MSF at the service level.
// For local development only:
if (isLocal(env)) {
    env.enableCheckpointing(3000);
}
```

### Anti-Pattern: Using CDC Source with Parallelism > 1 Without Server ID Range (MySQL)

```java
// AVOID: Single server ID with parallelism > 1
MySqlSource.builder()
    .serverId("5400")  // Only one ID for multiple parallel readers!
    ...

// CORRECT: Provide a range at least as large as the source parallelism
MySqlSource.builder()
    .serverId("5400-5404")  // Range of 5 IDs for up to 5 parallel readers
    ...
```

### Anti-Pattern: One Replication Slot Per Table (PostgreSQL)

```java
// AVOID: Creating separate CDC sources (and slots) for each table
PostgresIncrementalSource.<String>builder().tableList("public.orders").slotName("slot_orders").build();
PostgresIncrementalSource.<String>builder().tableList("public.customers").slotName("slot_customers").build();

// CORRECT: Use a single source with multiple tables sharing one slot
PostgresIncrementalSource.<String>builder()
    .tableList("public.orders", "public.customers", "public.products")
    .slotName("flink_cdc_slot")
    .build();
```

## Authentication

Database authentication for CDC sources on MSF must come from AWS Secrets Manager via in-application SDK lookup — see [Database Credentials and Secrets Management](#database-credentials-and-secrets-management) above for the supported pattern, required IAM, and TLS configuration. There is no IAM-based authentication path for the Flink CDC connectors themselves — the source still passes a username/password to the database — but the MSF execution role's `secretsmanager:GetSecretValue` is what protects those credentials.

Additional hardening that pairs with Secrets Manager:

- Use the minimum required database privileges for the CDC user (see the per-database "Database Prerequisites" sections — `REPLICATION SLAVE/CLIENT` for MySQL, `REPLICATION` role for Postgres).
- Enable TLS to the database. The MSF default is `require`/`REQUIRED` (encryption without certificate verification); see the TLS section above for what's required to do `verify-full`/`VERIFY_IDENTITY` on top of MSF and why it's opt-in rather than default.
- Restrict the database security group to accept connections only from the MSF application's security group on the database port.
- Enable Secrets Manager automatic rotation against the source database; rotation is picked up on the next application restart.

# Checkpoint Impact on Resources

## Overview

This guide covers how checkpointing affects Managed Service for Apache Flink application resources, including checkpoint size and memory consumption, frequency vs CPU/network trade-offs, checkpoint duration exceeding interval, and OOM/GC diagnostic steps.

For KPU sizing, operator parallelism tuning, and MSF configuration overrides, see [resource-optimization.md](resource-optimization.md).

## Checkpoint Size and KPU Memory Consumption

During a checkpoint, Flink snapshots operator state and uploads it to S3 (Managed Service for Apache Flink-managed bucket). This process consumes memory and CPU on each TaskManager:

- **RocksDB state backend** (default on Managed Service for Apache Flink): For incremental checkpoints, Flink uploads only new SST files created since the last checkpoint to S3. The async phase (uploading to S3) does not block record processing but does consume network bandwidth and some memory for upload buffers. The sync phase (snapshotting RocksDB) briefly blocks the subtask.
- Larger state per TaskManager means more data to read and upload, increasing memory and network pressure during the checkpoint window. Skewed state distribution can cause individual TaskManagers to become bottlenecks even when aggregate resources are sufficient.
- There is no official AWS or Flink guidance specifying a fixed percentage of KPU memory to reserve for checkpoint overhead. The actual impact depends on state size, checkpoint type (incremental vs full), and upload concurrency. Monitor `lastCheckpointDuration` and `heapMemoryUtilization` during checkpoint windows to assess whether checkpoint overhead is causing memory pressure.

## Checkpoint Frequency vs CPU and Network Bandwidth

Checkpoint frequency is configured via the Managed Service for Apache Flink `CheckpointInterval` setting (default: 60000ms / 60 seconds). The `MinPauseBetweenCheckpoints` (default: 5000ms) prevents continuous checkpointing when a checkpoint takes longer than the interval. You must set `ConfigurationType` to `CUSTOM` to modify these values. More frequent checkpoints:

- **Increase CPU usage**: each checkpoint triggers RocksDB file reads and S3 uploads across all TaskManagers. With incremental checkpoints (Managed Service for Apache Flink default), the CPU impact is proportional to state changes, not total state size.
- **Increase network bandwidth**: checkpoint data flows from TaskManagers to S3. With large state and frequent checkpoints, this can compete with data processing traffic.
- **Reduce recovery time**: more frequent checkpoints mean less data to replay from sources after a failure.

**Trade-off guidance per Flink docs:** When checkpoints frequently take longer than the base interval, the system ends up constantly taking checkpoints, tying up resources and reducing operator progress. Use `MinPauseBetweenCheckpoints` to prevent this. The Flink documentation does not prescribe specific interval ranges for state sizes — tune based on observed `lastCheckpointDuration` relative to your interval, and ensure checkpoints complete well within the interval with room to spare.

## Checkpoint Duration Exceeding Interval

**Symptoms:**

- `lastCheckpointDuration` in CloudWatch approaches or exceeds the configured checkpoint interval
- `numberOfInProgressCheckpoints` stays > 0 for extended periods
- Increasing `backPressuredTimeMsPerSecond` during checkpoint windows
- `millisBehindLatest` (Kinesis) or consumer lag (Kafka) grows during checkpoints

**Consequences:**

- By default, the next checkpoint is triggered immediately once the ongoing one completes. With `MinPauseBetweenCheckpoints` (default 5s on Managed Service for Apache Flink), there's a minimum gap, but the system can still end up constantly checkpointing.
- For aligned checkpoints (the default mode), checkpoint barriers can cause channels to block while waiting for alignment, contributing to backpressure. Unaligned checkpoints (available from Flink 1.15+, requestable via AWS support for Managed Service for Apache Flink) avoid this alignment delay but have other trade-offs.
- In extreme cases, checkpoint timeouts trigger checkpoint failures, and repeated failures can cause application restarts.

**Remediation:**

1. **Verify incremental checkpoints are active** (they are enabled by default on Managed Service for Apache Flink). If for some reason they were overridden, re-enable them — incremental checkpoints only upload state changes since the last checkpoint, dramatically reducing upload size for large state.
2. **Increase checkpoint interval** via the `UpdateApplication` API with `ConfigurationType: CUSTOM` to give more time for completion. Also consider increasing `MinPauseBetweenCheckpoints`.
3. **Add KPUs** to spread state across more TaskManagers, reducing per-TaskManager checkpoint size.
4. **Reduce state size**: add or tighten TTL on keyed state, reduce key cardinality, or use more compact serialization (POJO over Kryo).
5. **Request RocksDB tuning overrides** via AWS support if compaction or read amplification is the bottleneck.
6. **Consider buffer debloating** — request enablement via AWS support case. This can help applications with backpressure-related checkpoint issues.

## OOM and GC Diagnostic Steps

If the application throws `OutOfMemoryError` or shows sustained high GC activity:

1. **Check `heapMemoryUtilization` in CloudWatch.** If sustained > 80%, the application needs investigation and likely a scale-up (see [monitoring-and-metrics.md](monitoring-and-metrics.md) for graduated thresholds: healthy ≤ 75%, scale-up signal > 80% sustained, critical alarm > 90%).
2. **Check `lastCheckpointSize` and `lastCheckpointDuration`.** Large checkpoints consume significant heap during snapshot creation.
3. **Review state TTL configuration.** Missing or overly long TTL causes state to accumulate indefinitely.
4. **Check for Kryo serialization fallbacks.** Kryo uses more memory than POJO serialization. Look for log messages: `"Class ... cannot be used as a POJO type"`.
5. **Review operator state usage.** Use Flink Web UI to check state size per operator. Identify operators with disproportionately large state.

**If the root cause is legitimate memory pressure after optimization:**

- Request a JVM heap size increase via AWS support
- Consider increasing KPU count to spread state across more TaskManagers
- Request RocksDB block cache increase if state reads are the bottleneck (high cache miss rate)

## Flink Java Dependency Management

This document provides the complete Maven dependency configuration for Flink projects on Amazon Managed Service for Apache Flink. For new applications, default to Flink 2.2. For existing applications, use the user's current Flink version.

**For Kinesis-specific API usage and code examples**, see `kinesis-connector-guide.md`.

### Version-Specific Dependency Mapping

Not all connector versions are published for every Flink version. Use the correct combination from this table:

| Dependency | Flink 1.20 | Flink 2.2 | Notes |
|---|---|---|---|
| `flink.version` | `1.20.3` | `2.2.0` | Core Flink artifacts use this directly |
| `target.java.version` | `11` | `17` | Flink 2.x requires Java 17 minimum |
| `flink-connector-kafka` | `3.4.0-1.20` | `4.0.1-2.0` | Connector uses separate versioning scheme |
| `flink-connector-aws-kinesis-streams` | `5.1.0-1.20` | `6.0.0-2.0` | Connector uses separate versioning scheme |
| `flink-statebackend-rocksdb` | `1.20.3` (uses `flink.version`) | `2.2.0` (uses `flink.version`) | Available for both, can also use HashMap state backend via support case in MSF for apps that have small states that can fit in memory and benefit from lower latency state access |
| `aws-kinesisanalytics-runtime` | `1.2.0` | `1.2.0` | Same version for both |
| `aws-msk-iam-auth` | `2.3.5` | `2.3.5` | Version-agnostic |
| `maven-compiler-plugin` | `3.8.1` | `3.11.0` | Newer version recommended for Java 17 |

**Flink 2.2 additional notes:**

- `flink-statebackend-forst` is available as an alternative to `flink-statebackend-rocksdb` for disaggregated state management
- Logging uses `log4j-slf4j-impl` (same as 1.20; older Flink versions used `slf4j-log4j12`)
- See `flink-2x-migration.md` for API changes that affect application code (e.g., `open(OpenContext)`, `Duration` instead of `Time`, removed `SourceFunction`/`SinkFunction`)

### Example pom.xml for a Flink project

The example below defaults to Flink 2.2 properties (recommended for new applications). To use Flink 1.20 for existing applications, replace the properties block with the Flink 1.20 values from the table above.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xmlns="http://maven.apache.org/POM/4.0.0"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <!-- REPLACE: Use your application's package name -->
    <groupId>com.example</groupId>
    <!-- REPLACE: Use your application's artifact name -->
    <artifactId>my-flink-app</artifactId>
    <!-- Increment version as new changes are released -->
    <version>1.0</version>
    <packaging>jar</packaging>

    <!-- ============================================================
         Flink 2.2 properties (default for new applications)
         For Flink 1.20, replace with:
           target.java.version  = 11
           flink.version        = 1.20.3
           kafka.version        = 3.4.0-1.20
           kinesis-streams.version = 5.1.0-1.20
           maven.compiler.plugin.version = 3.8.1
         ============================================================ -->
    <properties>
        <!-- Build configs -->
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <buildDirectory>${project.basedir}/target</buildDirectory>
        <jar.finalName>${project.name}-${project.version}</jar.finalName>
        <target.java.version>17</target.java.version>
        <maven.compiler.source>${target.java.version}</maven.compiler.source>
        <maven.compiler.target>${target.java.version}</maven.compiler.target>

        <!-- Dependency versions -->
        <flink.version>2.2.0</flink.version>
        <kda.runtime.version>1.2.0</kda.runtime.version>
        <log4j.version>2.23.1</log4j.version>
        <!-- Not all kafka / flink version combinations are published on Maven -->
        <kafka.version>4.0.1-2.0</kafka.version>
        <msk-iam-auth.version>2.3.5</msk-iam-auth.version>
        <!-- Not all Kinesis / flink version combinations are published on Maven -->
        <kinesis-streams.version>6.0.0-2.0</kinesis-streams.version>
        <maven.compiler.plugin.version>3.11.0</maven.compiler.plugin.version>
    </properties>

    <!-- Java SDK for AWS required in general -->
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>com.amazonaws</groupId>
                <artifactId>aws-java-sdk-bom</artifactId>
                <version>1.12.677</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <!-- Apache Flink dependencies required for most projects -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-clients</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-runtime-web</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-base</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>

        <!-- Flink Table Runtime — required when application code references
             internal Table Runtime types (InternalTypeInfo, RowDataSerializer,
             internal RowData converters). Commonly needed by Iceberg sinks
             that emit/consume RowData on a side output. Provided by MSF at
             runtime.

             Note: bridge != runtime. flink-table-api-java-bridge gives you
             the public Table API/SQL surface (DDL, TableEnvironment,
             DataStream<->Table conversion). flink-table-runtime gives you
             the internal data-structure types referenced by sinks and
             custom serializers. -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-runtime</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>

        <!-- Managed Service for Apache Flink Runtime required for all projects -->
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-kinesisanalytics-runtime</artifactId>
            <version>${kda.runtime.version}</version>
            <scope>provided</scope>
        </dependency>

        <!-- Kafka Connector required for Kafka source/sink projects -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-kafka</artifactId>
            <version>${kafka.version}</version>
        </dependency>
        <!-- MSK IAM Auth required for Kafka projects using MSK IAM -->
        <dependency>
            <groupId>software.amazon.msk</groupId>
            <artifactId>aws-msk-iam-auth</artifactId>
            <version>${msk-iam-auth.version}</version>
        </dependency>

        <!-- Kinesis connector for Kinesis Streams source/sink projects -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-aws-kinesis-streams</artifactId>
            <version>${kinesis-streams.version}</version>
        </dependency>


        <!-- RocksDB State Backend required for all projects -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-statebackend-rocksdb</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <!-- JSON Processing required for JSON projects -->
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.15.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-json</artifactId>
            <version>${flink.version}</version>
        </dependency>


        <!-- Logging - generally required -->
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-slf4j-impl</artifactId>
            <version>${log4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
            <version>${log4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>${log4j.version}</version>
        </dependency>
    </dependencies>


    <!-- Profile for local testing - includes provided dependencies -->
    <profiles>
        <profile>
            <id>local</id>
            <dependencies>
                <dependency>
                    <groupId>org.apache.flink</groupId>
                    <artifactId>flink-clients</artifactId>
                    <version>${flink.version}</version>
                    <scope>compile</scope>
                </dependency>
                <dependency>
                    <groupId>org.apache.flink</groupId>
                    <artifactId>flink-streaming-java</artifactId>
                    <version>${flink.version}</version>
                    <scope>compile</scope>
                </dependency>
                <dependency>
                    <groupId>org.apache.flink</groupId>
                    <artifactId>flink-runtime-web</artifactId>
                    <version>${flink.version}</version>
                    <scope>compile</scope>
                </dependency>
                <dependency>
                    <groupId>org.apache.flink</groupId>
                    <artifactId>flink-connector-base</artifactId>
                    <version>${flink.version}</version>
                    <scope>compile</scope>
                </dependency>

                <!-- Managed Service for Apache Flink Runtime -->
                <dependency>
                    <groupId>com.amazonaws</groupId>
                    <artifactId>aws-kinesisanalytics-runtime</artifactId>
                    <version>${kda.runtime.version}</version>
                    <scope>compile</scope>
                </dependency>
            </dependencies>
        </profile>
    </profiles>

    <!-- Most of the below is boilerplate for managing shaded JARs -->
    <build>
        <directory>${buildDirectory}</directory>
        <finalName>${jar.finalName}</finalName>
        <plugins>
            <!-- Java Compiler -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${maven.compiler.plugin.version}</version>
                <configuration>
                    <source>${target.java.version}</source>
                    <target>${target.java.version}</target>
                </configuration>
            </plugin>
            <!-- Maven Shade Plugin -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.1</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <artifactSet>
                                <excludes>
                                    <exclude>org.apache.flink:force-shading</exclude>
                                    <exclude>com.google.code.findbugs:jsr305</exclude>
                                </excludes>
                            </artifactSet>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                            <transformers>
                                <transformer
                                        implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                                <transformer
                                        implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <!-- REPLACE: Use your application's fully qualified main class -->
                                    <mainClass>com.example.MyFlinkJob</mainClass>
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>
```

# Managed Service for Apache Flink Development Environment Setup

## Overview

This guide provides setup instructions for Apache Flink development targeting Amazon Managed Service for Apache Flink deployment. The setup ensures compatibility with Managed Service for Apache Flink requirements while enabling efficient local development and testing workflows using Docker containers that mirror the Managed Service for Apache Flink environment. For new applications, default to Flink 2.2. For existing applications, use the user's current Flink version.

## Prerequisites

Before starting, ensure you have:

- Administrative access to your development machine
- Stable internet connection for downloading dependencies
- At least 16GB RAM and 30GB free disk space (for Docker containers)
- Docker Desktop installed and running
- Java 11 (for Flink 1.20) or Java 17 (for Flink 2.2) installed
- Basic familiarity with command-line operations and Docker concepts

## Core Development Environment

### 1. Docker Desktop Installation

Docker is required for running Flink, Kafka, LocalStack, and other infrastructure locally for testing. Ensure Docker is installed.

**Verification:**

```bash
docker --version
docker-compose --version
docker run hello-world
```

### 2. Java Development Kit (JDK) for Container Development

While Flink runs in containers, JDK is still needed for compilation and IDE support. Flink 1.20 requires Java 11. Flink 2.2 requires Java 17 (Java 21 also supported).

**Verification:**

```bash
java -version
javac -version
echo $JAVA_HOME
```

## Docker-Based Flink Development Environment

### Docker Compose Configuration

Create a comprehensive Docker Compose setup that mirrors the Managed Service for Apache Flink environment. Adjust the Flink image tag to match your target version:

- Flink 1.20: `flink:1.20-java11`
- Flink 2.2: `flink:2.2-java17`

**Create `docker-compose.yml` in your project root:**

```yaml
version: '3.8'

# Set FLINK_IMAGE_TAG in your environment or .env file:
#   Flink 1.20: FLINK_IMAGE_TAG=1.20-java11
#   Flink 2.2:  FLINK_IMAGE_TAG=2.2-java17

services:
  # Flink JobManager
  jobmanager:
    image: flink:${FLINK_IMAGE_TAG:-2.2-java17}
    hostname: jobmanager
    container_name: flink-jobmanager
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        jobmanager.memory.process.size: 1600m
        jobmanager.execution.failover-strategy: region
      - RUNTIME_ENVIRONMENT=local
    volumes:
      - ./flink-jobs:/opt/flink/jobs
      - flink-checkpoints:/tmp/flink-checkpoints
      - flink-savepoints:/tmp/flink-savepoints
    networks:
      - flink-network

  # Flink TaskManager
  taskmanager:
    image: flink:${FLINK_IMAGE_TAG:-2.2-java17}
    depends_on:
      - jobmanager
    command: taskmanager
    scale: 2
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        taskmanager.numberOfTaskSlots: 2
        parallelism.default: 2        
        taskmanager.memory.process.size: 1728m
        taskmanager.memory.managed.fraction: 0.4
        state.backend: rocksdb
        state.checkpoints.dir: file:///tmp/flink-checkpoints
        state.savepoints.dir: file:///tmp/flink-savepoints
        execution.checkpointing.interval: 60000
        execution.checkpointing.mode: EXACTLY_ONCE
        restart-strategy: exponential-delay
        restart-strategy.exponential-delay.initial-backoff: 10s
        restart-strategy.exponential-delay.max-backoff: 2min
        restart-strategy.exponential-delay.backoff-multiplier: 2.0
      - RUNTIME_ENVIRONMENT=local
    volumes:
      - ./flink-jobs:/opt/flink/jobs
      - flink-checkpoints:/tmp/flink-checkpoints
      - flink-savepoints:/tmp/flink-savepoints
    networks:
      - flink-network

  # Kafka (for streaming data sources)
  zookeeper:
    image: confluentinc/cp-zookeeper:7.4.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    networks:
      - flink-network

  kafka:
    image: confluentinc/cp-kafka:7.4.0
    hostname: kafka
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost
    networks:
      - flink-network

  # LocalStack (for local AWS service emulation: Kinesis, S3, DynamoDB, etc.)
  localstack:
    image: localstack/localstack:3.5.0
    container_name: localstack
    ports:
      - "4566:4566"
    environment:
      SERVICES: kinesis,s3,dynamodb,cloudwatch
      DEFAULT_REGION: us-east-1
    networks:
      - flink-network

volumes:
  flink-checkpoints:
  flink-savepoints:

networks:
  flink-network:
    driver: bridge
```

### Starting the Development Environment

**1. Create Project Structure:**

```bash
# Create Managed Service for Apache Flink project directory
mkdir my-msf-app
cd my-msf-app

# Create required directories
mkdir -p flink-jobs src/main/java src/test/java

# Copy docker-compose.yml to project root
```

**2. Start Docker Environment:**

```bash
# Start all services
docker-compose up -d

# Verify services are running
docker-compose ps

# Check taskmanager logs for submitted Flink jobs 
docker-compose logs taskmanager

# Check Flink Web UI
open http://localhost:8081

# Check Kafka
docker-compose exec kafka kafka-topics --bootstrap-server localhost:9092 --list
```

## Build Tools and Dependencies

### Maven Project Template for Managed Service for Apache Flink

Build the JAR in local mode to include Managed Service for Apache Flink provided dependencies:

```bash
# For Flink 1.20 (Java 11):
mvn clean package -Plocal -q

# For Flink 2.2 (Java 17):
# Set JAVA_HOME to your Java 17 installation path before building
export JAVA_HOME=<path-to-java-17>
mvn clean package -Plocal -q

cp target/my-flink-app-1.0.jar flink-jobs/

docker exec flink-jobmanager flink run /opt/flink/jobs/my-flink-app-1.0.jar
```

See `dependency-management.md` for examples of dependencies and pom.xml requried for Managed Service for Apache Flink.

### Managed Service for Apache Flink-Specific Tools

**AWS CLI Managed Service for Apache Flink Commands:**

```bash
# List Managed Service for Apache Flink applications
aws kinesisanalyticsv2 list-applications

# Describe Managed Service for Apache Flink application
aws kinesisanalyticsv2 describe-application --application-name my-app

# Create savepoint
aws kinesisanalyticsv2 create-application-snapshot --application-name my-app --snapshot-name my-snapshot
```

# First-Fault Isolation for Restart Loops

## Overview

Diagnose Flink jobs that are restarting, crashing, or stuck in a failure loop. The current exception in recent logs is almost always a **loop-sustaining side effect**, not the root cause. Find the FIRST task failure (attempt #0) and read its `throwableInformation`.

Use this guide when `fullRestarts > 0`, `downtime > 0`, the Flink job keeps cycling RUNNING → STARTING → RUNNING, or recent logs show repeating exceptions without obvious cause.

## #1 Rule

When a Flink job is in a restart loop, recent logs show loop iterations, NOT the root cause. Everything else — `addSplitsBack`, `UnsupportedOperationException: Partial recovery is not supported`, SSLException during restarts, connection errors after a stop — is a **loop-sustaining side effect**. The original trigger is in attempt #0's `throwableInformation` field.

## Methodology

### Step 1: Find crash onset (binary search)

Restart loops can run for days or weeks. The "last 1h" log window will only show loop iterations. Find the hour where `fullRestarts` first transitioned from 0 → >0 by binary searching backward across 14 days using CloudWatch:

```bash
for DAYS_AGO in 14 7 3 1; do
  aws cloudwatch get-metric-statistics --namespace AWS/KinesisAnalytics \
    --metric-name fullRestarts --dimensions Name=Application,Value="$APP" \
    --start-time $(date -u -d "${DAYS_AGO} days ago" +%Y-%m-%dT%H:%M:%SZ) \
    --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) --period 3600 --statistics Maximum
done
```

Halve the window until you isolate the **hour** where `fullRestarts` went from 0 to >0. If `fullRestarts > 0` for the entire 14-day window, the job has never been stable on the current configuration.

In Flink 2.2+, use `numRestarts` (cumulative counter) — `fullRestarts` is removed.

### Step 2: Find the last successful checkpoint or transition to RUNNING before crash onset

If the app has checkpointing enabled, you can look for the last time the application successfully took a checkpoint:

```bash
aws logs filter-log-events --log-group-name "/aws/kinesis-analytics/$APP" \
  --filter-pattern '"Completed checkpoint"' \
  --start-time $((CRASH_ONSET_MS - 3600000)) --end-time $CRASH_ONSET_MS \
  --limit 50 --output json | jq '.events[-1]'
```

This gives the timestamp where the job was last healthy.

For apps without checkpointing, or using unaligned checkpointing, it is more reliable to identify the first time the app transitioned from a RUNNING state (see step 3).

### Step 3: Find attempt #0 — the FIRST task failure

In the 30 seconds after the last successful checkpoint, fetch ALL log events (do not filter) and look for:

- `"switched from RUNNING to"` — the task status change event
- `"#0 ("` in the task name — `#0` confirms attempt zero (first failure, not a loop iteration)
- `"Recovering subtask"` followed by `"Triggering job failover"` — the restart loop beginning

```bash
aws logs filter-log-events --log-group-name "/aws/kinesis-analytics/$APP" \
  --start-time $LAST_CHECKPOINT_MS --end-time $((LAST_CHECKPOINT_MS + 30000)) \
  --limit 50 --output json
```

### Step 4: Extract `throwableInformation` from attempt #0

MSF emits structured JSON logs. The actual exception and stack trace are in the `throwableInformation` field, NOT the `message` field. CloudWatch `filter-log-events` searches the raw log string, so a pattern like `"JsonParseException"` may not match if the exception is nested inside `throwableInformation`.

The reliable approach is to filter on `"switched from RUNNING"` then extract the field with jq:

```bash
aws logs filter-log-events --log-group-name "/aws/kinesis-analytics/$APP" \
  --filter-pattern '"switched from RUNNING"' \
  --start-time $FIRST_FAILURE_MS --end-time $((FIRST_FAILURE_MS + 2000)) \
  --output json | jq '.events[].message | fromjson | .throwableInformation'
```

Follow the `Caused by` chain downward, but **the deepest exception is not automatically the root cause** — it's just where the stack trace ends. Often it is the root cause (a `JsonParseException` at a specific offset, an `AmazonS3Exception: 403`, a user-code `NullPointerException` at a named line). Sometimes it isn't. Treat the deepest exception as the root cause only when it has **specific, actionable context** — a class.method() in user code, a connector message naming a resource and HTTP status, a parser pointing at a byte offset.

#### When the deepest exception is itself a symptom

Some exceptions terminate the stack trace but tell you almost nothing about *why*. They mean "the TaskManager / job stopped responding," not "here is what went wrong." When you see one of these as the deepest `Caused by`, **stop reading stack traces and pivot to metrics and live diagnostics** — the cause is no longer in the logs, it's in what was happening to the JVM or the cluster at that moment.

| Exception in `throwableInformation` | What it actually means | Where to pivot |
|---|---|---|
| `TimeoutException: Heartbeat of TaskManager with id ... timed out` | TM stopped sending heartbeats — GC pause, OOMKill, network partition, or CPU starvation | CloudWatch `heapMemoryUtilization`, `oldGenerationGCTime`, `cpuUtilization`, `containerCPUUtilization` for ±5 min around `FIRST_FAILURE_MS`; thread dump if app is still RUNNING |
| `Connection unexpectedly closed by remote task manager` / `LostTaskException` | Peer TM disappeared mid-shuffle | Same metrics on the *peer* TM; check whether a TM was killed by the resource manager (look for `TaskManager exited` / container exit codes in logs) |
| `JobMasterException: ... has not been heard from` | JobManager lost contact with TMs | JM-side GC and CPU; ZooKeeper / leader-election logs |
| `OutOfMemoryError: Java heap space` *with no application frames in the trace* | Heap exhausted during framework op (checkpoint serialization, network buffers) | `heapMemoryUtilization`, checkpoint size growth, state-backend choice; not user-code unless app frames are in the trace |
| `IOException: Insufficient number of network buffers` | Buffer pool exhausted | parallelism × shuffle-edge fan-out and `taskmanager.network.memory.*` config |
| Generic `IOException` / `ExecutionException` wrapping nothing specific | Wrapper with no diagnostic content | Keep walking `Caused by`; if the wrapper *is* the deepest entry, the cause is upstream of the JVM and you need metrics, not logs |

The pivot itself: re-anchor on `FIRST_FAILURE_MS` and pull a tight window of metrics and JM/TM logs around it.

```bash
# Heap, GC, CPU at the moment of failure (±5 min)
START=$(( (FIRST_FAILURE_MS - 300000) / 1000 ))
END=$(( (FIRST_FAILURE_MS + 300000) / 1000 ))
for METRIC in heapMemoryUtilization oldGenerationGCTime cpuUtilization containerCPUUtilization; do
  aws cloudwatch get-metric-statistics --namespace AWS/KinesisAnalytics \
    --metric-name "$METRIC" --dimensions Name=Application,Value="$APP" \
    --start-time @$START --end-time @$END --period 60 \
    --statistics Maximum Average
done

# Was a TaskManager killed (OOMKill, container exit) just before the heartbeat timeout?
aws logs filter-log-events --log-group-name "/aws/kinesis-analytics/$APP" \
  --filter-pattern '?"TaskManager exited" ?"OutOfMemory" ?"killed by" ?"exit code"' \
  --start-time $((FIRST_FAILURE_MS - 120000)) --end-time $FIRST_FAILURE_MS
```

Concrete pattern to look for: `oldGenerationGCTime` rising sharply or `heapMemoryUtilization` near 100% in the minutes before a heartbeat timeout = GC pressure / impending OOM, not a network issue. `containerCPUUtilization` pinned at the limit before a heartbeat timeout = CPU starvation, also not a network issue. If neither is true, suspect a real network event (VPC issue, ENI exhaustion) and check the EC2/VPC side.

Only after you have a coherent story across logs *and* metrics is it safe to call the root cause. If you can't get there from the data, say so explicitly in the report rather than promoting a symptom.

## Loop-Sustainer vs Original Trigger

| Exception | Role |
|-----------|------|
| Found in attempt #0's `throwableInformation` with specific context | **ORIGINAL TRIGGER** (root cause) |
| Heartbeat / `TaskManagerLost` / framework `OutOfMemoryError` as deepest `Caused by` | Symptom of GC / OOMKill / CPU / network — **pivot to metrics**, not the deepest cause |
| `UnsupportedOperationException: Partial recovery is not supported` | Loop sustainer (Kinesis Source FLIP-27 limitation) |
| `addSplitsBack` in stack trace | Loop sustainer (connector cannot partial-recover) |
| `SSLException`, `ConnectException` during restart | Loop side effect (rapid restarts cause connection churn) |
| `Recovering subtask ... Triggering job failover` | Loop iteration mechanics |
| `Restarting job` / `Task failed` | Symptoms — never report as root cause |
| `Caused by: <wrapper>` with no specific context | Follow chain deeper |

## Kinesis Source FLIP-27 Recovery Blocker

The `KinesisStreamsSource` (FLIP-27) does not support partial recovery. Any task failure with this source becomes an infinite restart loop because every restart attempt fails with `UnsupportedOperationException` from `addSplitsBack()`. The original cause in attempt #0 may have been transient (a single bad record, a momentary network blip), but the loop is permanent until you escape it.

**Escape procedure** (causes reprocessing or data loss — confirm with user):

1. Fix the root cause from attempt #0
2. Stop the application with `--force`
3. Poll until READY
4. Start with `ApplicationRestoreType=SKIP_RESTORE_FROM_SNAPSHOT`

## Diagnostic Anchors (Stack Trace Reading)

| Stack Frame | What It Means |
|-------------|---------------|
| `at com.<your-package>.MyUDF.process()` | User code bug |
| `at ...JsonDeserializationSchema.deserialize()` | Malformed input data |
| `at ...KafkaConsumer.poll()` / `at ...KinesisProxy...` | Connector/infrastructure issue (read message carefully) |
| `at org.apache.flink.runtime...` only | Framework-level — usually triggered by deeper user/connector cause; keep digging |
| Bare `IOException` wrapping deeper exception | Follow `Caused by` |

## Flink Dashboard for Live Diagnosis

When the Flink job is running (not stuck restarting), use the Flink Dashboard REST API for thread dumps, per-vertex backpressure, and execution plan analysis. Get a pre-signed URL:

```bash
aws kinesisanalyticsv2 create-application-presigned-url \
  --application-name "$APP" --url-type FLINK_DASHBOARD_URL \
  --query 'AuthorizedUrl' --output text
```

Authenticate by curling the URL with `-c cookie.jar -L`, then use `${BASE}flinkdashboard/` as the API base. Useful endpoints:

| Endpoint | Purpose |
|----------|---------|
| `overview` | Cluster status, slots, jobs |
| `jobs/$JOB_ID` | Per-vertex parallelism, status, read/write counts |
| `jobs/$JOB_ID/checkpoints` | Checkpoint history with sizes and durations |
| `jobs/$JOB_ID/vertices/$VERTEX_ID/backpressure` | Per-vertex backpressure level |
| `jobs/$JOB_ID/vertices/$VERTEX_ID/subtasks` | Per-subtask metrics — **detect data skew** |
| `jobs/$JOB_ID/plan` | Execution DAG with ship strategies |
| `taskmanagers/$TM_ID/thread-dump` | JVM thread dump for hot-thread / lock-contention analysis |

`create-application-presigned-url` fails on non-RUNNING applications.

### Thread Dump Patterns

| Stack Pattern | Diagnosis |
|--------------|-----------|
| `BLOCKED` on `synchronized` | Lock contention — reduce shared state, use async I/O |
| `TIMED_WAITING` in `Thread.sleep` | Explicit sleep in user code |
| `WAITING` in `Object.wait` on mailbox | Operator idle (normal for low-throughput) |
| `RUNNABLE` in `RocksDB.*` methods | State backend I/O — increase managed memory or enable incremental checkpoints |
| `RUNNABLE` in `SocketInputStream.read` | Network I/O — check source/sink latency |
| `RUNNABLE` in user package | CPU-bound user code |
| Multiple threads in same method | Hot method — profile and optimize |

### Job Graph Anti-Patterns (from `plan` and `subtasks` endpoints)

| Anti-Pattern | Detection | Fix |
|-------------|-----------|-----|
| Excessive shuffles | Multiple `HASH`/`REBALANCE` `ship_strategy` between vertices | Chain operators, use `forward` partitioning |
| Data skew | Subtask `read-records` variance > 5× across the same vertex | `rebalance()` instead of `keyBy()` on hot keys; pre-split |
| Late filtering | `read:write` ratio > 100:1 | Push filters upstream (predicate pushdown) |
| Parallelism mismatch | Downstream parallelism < upstream | Match parallelism across connected operators |
| Unchained operators | `FORWARD` strategy across separate vertices | Same parallelism + remove `disableChaining()` |

## Mandatory Report Format

When delivering diagnosis to the user, structure it as:

```
## Root Cause
First fault at <timestamp>: <specific exception> at <class.method>
Last healthy: <timestamp of last successful checkpoint>
Trigger: <one sentence — bad record, OOM, schema change, etc.>

## Symptoms (cascading from root cause)
- <symptom 1> — <relationship to root cause>
- <symptom 2> — <relationship to root cause>

## Fix
<specific action; not "investigate further">
```

State transitions (`RUNNING → FAILED`, `Task failed`, `Restarting job`) are **always symptoms**. Never report a state transition as a root cause.

## Common Mistakes

| Mistake | Why It Fails |
|---------|--------------|
| Searching last 1–24h on a multi-day restart loop | You find loop iterations, not the original failure |
| Reporting `addSplitsBack` / `UnsupportedOperationException` as root cause | These are loop-sustaining side effects of FLIP-27 Kinesis source |
| Reading only the `message` field of MSF JSON logs | Stack trace lives in `throwableInformation` |
| Reporting the deepest `Caused by` as root cause when it's a heartbeat timeout, `TaskManagerLost`, or framework `OutOfMemoryError` | Those are TM-died symptoms — pivot to heap/GC/CPU metrics around `FIRST_FAILURE_MS` |
| Filtering CloudWatch logs by exception class name | Filter searches raw string; nested fields may not match — filter on the `RUNNING to FAILED` event instead |
| Concluding from truncated `filter-log-events` output | If `NextToken` is present or `--limit` was hit, requery with wider window |
| Calling `create-application-presigned-url` on a non-RUNNING app | Fails — only works while RUNNING |

## References

- [MSF Troubleshooting Guide](https://docs.aws.amazon.com/managed-flink/latest/java/troubleshooting.html)
- [Flink REST API](https://nightlies.apache.org/flink/flink-docs-stable/docs/ops/rest_api/)
- [FLIP-27: Refactor Source Interface](https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface)

# Flink 2.x Migration Guide

## Overview

This guide covers migrating Flink applications from 1.x to 2.x. Key changes: Java 17 minimum (Java 8 and 11 no longer supported), major API removals, and serialization breaking changes affecting state compatibility.

**CRITICAL**: State migration from 1.x to 2.x fails for applications using Kryo or POJOs with collection fields. See [State Compatibility](#state-compatibility) before planning migrations.

## Major Breaking Changes Summary

| Category | Change | Impact |
|----------|--------|--------|
| Java | Java 8 and 11 removed, Java 17 default, Java 21 experimental (not supported in MSF) | Must upgrade runtime |
| Source API | `SourceFunction` removed | Must migrate to new Source API |
| Sink API | `SinkFunction`, `SinkV1` removed | Must migrate to Sink V2 API |
| Config | `flink-conf.yaml` removed | Must use `config.yaml` (standard YAML) |
| Time API | `Time` class deprecated | Use `java.time.Duration` |
| Serialization | Kryo 2.x → 5.x, new collection serializers | State incompatibility (see below) |
| DataSet API | Entire DataSet API removed | Migrate to DataStream or Table API/SQL |
| Scala API | Scala API removed entirely | Use Java API (callable from Scala) |
| Python | Python 3.8 removed, Python 3.12 default | Update Python runtime |
| DataStream | `IterativeStream`, `TimeCharacteristic` removed | Refactor required |

## Dependency Changes

### Version Updates Required

```xml
<properties>
    <!-- Core versions -->
    <flink.version>2.2.0</flink.version>
    <target.java.version>17</target.java.version>
    <maven.compiler.source>17</maven.compiler.source>
    <maven.compiler.target>17</maven.compiler.target>
    
    <!-- Connector versions (use 2.x compatible) -->
    <kafka.version>4.0.1-2.0</kafka.version>
    <kinesis-streams.version>6.0.0-2.0</kinesis-streams.version>
    
    <!-- Updated dependencies -->
    <msk-iam-auth.version>2.3.5</msk-iam-auth.version>
    <aws.sdk.version>1.12.677</aws.sdk.version>
    <jackson.version>2.15.2</jackson.version>
    <lombok.version>1.18.36</lombok.version>
    <log4j.version>2.23.1</log4j.version>
    <maven.compiler.plugin.version>3.11.0</maven.compiler.plugin.version>
</properties>
```

### Logging Changes

Flink 2.x uses `log4j-slf4j-impl` instead of `slf4j-log4j12`.

> **Note:** If migrating from Flink 1.20, you likely already use `log4j-slf4j-impl`. This change only applies when migrating from Flink versions older than ~1.15 that used `slf4j-log4j12`.

```xml
<!-- Remove these -->
<!-- <artifactId>slf4j-log4j12</artifactId> -->

<!-- Use these -->
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-slf4j-impl</artifactId>
    <version>${log4j.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-api</artifactId>
    <version>${log4j.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>${log4j.version}</version>
</dependency>
```

### Glue Schema Registry Conflict

Exclude old flink-avro from Glue Schema Registry until it is updated for Flink 2.x:

```xml
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-avro</artifactId>
    <version>${flink.version}</version>
</dependency>
<dependency>
    <groupId>software.amazon.glue</groupId>
    <artifactId>schema-registry-flink-serde</artifactId>
    <version>1.1.15</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-avro</artifactId>
        </exclusion>
    </exclusions>
</dependency>
```

### Scope Changes for Standalone Deployment

For non-Managed Service for Apache Flink deployment, change scope from `provided` to `compile`:

```xml
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-streaming-java</artifactId>
    <version>${flink.version}</version>
    <scope>compile</scope> <!-- Required for CEP serialization in 2.x -->
</dependency>
```

## Code Changes

### Time API Migration

Replace `org.apache.flink.streaming.api.windowing.time.Time` with `java.time.Duration`:

```java
// Before (1.x)
import org.apache.flink.streaming.api.windowing.time.Time;
.window(TumblingProcessingTimeWindows.of(Time.seconds(10)))
.window(SlidingProcessingTimeWindows.of(Time.minutes(1), Time.seconds(1)))
.window(EventTimeSessionWindows.withGap(Time.seconds(30)))
.within(Time.seconds(10))

// After (2.x)
import java.time.Duration;
.window(TumblingProcessingTimeWindows.of(Duration.ofSeconds(10)))
.window(SlidingProcessingTimeWindows.of(Duration.ofMinutes(1), Duration.ofSeconds(1)))
.window(EventTimeSessionWindows.withGap(Duration.ofSeconds(30)))
.within(Duration.ofSeconds(10))
```

### Configuration API Migration

Replace string-based config with type-safe ConfigOptions:

```java
// Before (1.x)
config.setInteger("rest.port", 8081);
config.setBoolean("web.submit.enable", true);

// After (2.x)
import org.apache.flink.configuration.RestOptions;
import org.apache.flink.configuration.WebOptions;
config.set(RestOptions.PORT, 8081);
config.set(WebOptions.SUBMIT_ENABLE, true);
```

### Function Lifecycle Changes

`open()` method signature changed from `Configuration` to `OpenContext`:

```java
// Before (1.x)
@Override
public void open(Configuration parameters) {
    // initialization
}

// After (2.x)
@Override
public void open(org.apache.flink.api.common.functions.OpenContext openContext) throws Exception {
    // initialization
}
```

This change applies to every `RichFunction` subclass — `RichMapFunction`, `RichFlatMapFunction`, `RichFilterFunction`, `KeyedProcessFunction`, `BroadcastProcessFunction`, `KeyedBroadcastProcessFunction`, `ProcessWindowFunction`, async I/O `RichAsyncFunction`, etc. Any code that overrides `open(Configuration)` will fail to compile against Flink 2.2. `OpenContext` does not carry the legacy `Configuration` key/value bag — read runtime properties via the `KinesisAnalyticsRuntime.getApplicationProperties()` flow or pass them through your function's constructor.

The `open()` change is one of several Flink 2.x breaking API changes you'll likely hit during the same migration. See [Removed APIs and Migration Paths](#removed-apis-and-migration-paths) for the full table; the headline removals are:

- `SourceFunction` and `SinkFunction` are removed in favor of `Source` (FLIP-27) and `Sink` (FLIP-143) — `env.addSource()` / `stream.addSink()` no longer compile.
- `org.apache.flink.api.common.time.Time` is deprecated in favor of `java.time.Duration`. Anything that took `Time` (TTL, idleness, async I/O timeout) now takes `Duration`.
- `TimeCharacteristic` is removed — event-time is the only mode and `setStreamTimeCharacteristic()` is gone.
- `enableForceAvro()` and the convenience Kryo registration methods on `StreamExecutionEnvironment` are removed; use `env.getConfig()` equivalents.

### CEP Pattern Type Information

CEP requires explicit TypeInformation for pattern output:

```java
// Before (1.x)
CEP.pattern(stream, pattern)
    .inEventTime()
    .select(this::extractResult);

// After (2.x)
import org.apache.flink.api.common.typeinfo.TypeHint;
import org.apache.flink.api.common.typeinfo.TypeInformation;

CEP.pattern(stream, pattern)
    .inEventTime()
    .select(
        this::extractResult,
        TypeInformation.of(new TypeHint<ResultType>() {})
    );
```

### POJO Requirements

POJOs must implement Serializable with no-args constructor:

```java
// Before (1.x) - might work without these
@Data
@Builder
public class Event {
    private String id;
    private long timestamp;
}

// After (2.x) - required for proper serialization
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class Event implements Serializable {
    private static final long serialVersionUID = 1L;
    private String id;
    private long timestamp;
}
```

### Removed Classes

- `org.apache.flink.streaming.api.TimeCharacteristic` - removed, event time is default
- `org.apache.flink.api.java.typeutils.runtime.kryo.Serializers` - Kryo serializer classes removed
- `flink-java` module removed entirely

## State Compatibility

### Breaking Changes Summary

Three serialization incompatibilities prevent state migration from 1.x to 2.x:

| Issue | Root Cause | Affected Patterns | Error Signature |
|-------|-----------|-------------------|-----------------|
| Kryo reference tracking | Kryo 2.x → 5.x upgrade | `registerTypeWithKryoSerializer()` | `IndexOutOfBoundsException: Index 116 out of bounds for length 1` |
| Kryo CollectionSerializer | Kryo's internal collection format changed | Generic collections: `List<T>`, `Map<K,V>`, `Set<T>` in state | `ClassNotFoundException: value2` (data misinterpreted as class names) |
| PojoSerializer collection handling | TypeExtractor selects different serializers (FLINK-34037) | POJO fields: List, Map, Set, Collection, Queue, Deque | `StateMigrationException: PojoSerializer@8bf85b5d incompatible with @3282ee3` |

### Compatible Patterns (State Migrates Successfully)

**Serialization Methods:**

- **Avro serialization** with explicit `AvroTypeInfo` - schema-based, independent of Flink's type system
- **Protobuf serialization** - schema-based, bypasses TypeExtractor
- **Custom TypeSerializer** implementations - user-controlled serialization
- **Simple POJOs** without collection fields - no TypeExtractor collection handling
- **Flink Tuples** - direct field access, no reflection
- **Primitive types** - fastest, no serialization changes

**State Types (all compatible with above serializers):**

- ValueState, MapState, ListState, ReducingState, AggregatingState
- BroadcastState with control streams
- Operator state (even-split and union redistribution)
- Window state (tumbling, sliding, session windows)
- Timer state (event-time and processing-time)

**Connectors:**

- Kinesis connector state (v5.0+ only — see note below; default polling and Enhanced Fan-Out)
- Kafka connector state (offsets, partition tracking)

**CRITICAL — Kinesis Connector Version Prerequisite:** KDS connector versions below 5.0 maintain state that is incompatible with the Flink 2.2 Kinesis connector (v6.0.0-2.0). You must migrate to connector v5.0+ on Flink 1.x before upgrading to Flink 2.x. See `kinesis-connector-guide.md` for migration paths and the [AWS blog post](https://aws.amazon.com/blogs/big-data/introducing-the-new-amazon-kinesis-source-connector-for-apache-flink/) for details.

**Table API/SQL (with caveat):**

- All tested patterns compatible: GROUP BY, window aggregations (TUMBLE, HOP, SESSION)
- Stream joins (INNER, LEFT OUTER), Top-N, deduplication
- DISTINCT aggregations, OVER windows
- Table API provides alternative migration path avoiding DataStream serialization issues
- **Caveat:** Apache Flink does not guarantee state compatibility between major versions for Table API applications. Always test in a non-production environment first.

### Incompatible Patterns (State Migration Fails)

**Direct Kryo Usage:**

```java
// INCOMPATIBLE - Kryo 2.x → 5.x reference tracking changed
env.getConfig().registerTypeWithKryoSerializer(MyType.class, MyKryoSerializer.class);
```

**Generic Collections in State:**

```java
// INCOMPATIBLE - Kryo CollectionSerializer format changed
ValueState<List<String>> listState;
ValueState<Map<String, Integer>> mapState;
ValueState<Set<Long>> setState;
```

**POJOs with Collection Fields:**

```java
// INCOMPATIBLE - TypeExtractor selects different collection serializers
public class UserSession {
    public String userId;
    public List<String> eventTypes;        // BREAKS
    public Map<String, Integer> counts;    // BREAKS
    public Set<String> visitedPages;       // BREAKS
    public Queue<Event> eventQueue;        // BREAKS
}
```

**Scala Case Classes:**

```scala
// INCOMPATIBLE - Serialized via Kryo in Flink 1.x, Kryo v2→v5 binary format change breaks state
case class UserEvent(userId: String, eventType: String, timestamp: Long)
```

**Java Records:**

```java
// INCOMPATIBLE - Typically fall back to Kryo serialization in Flink 1.x
// Verify by testing with env.getConfig().disableGenericTypes()
public record UserEvent(String userId, String eventType, long timestamp) {}
```

**Third-Party Library Types:**

```java
// INCOMPATIBLE - Types without a registered custom serializer fall back to Kryo
// The Kryo v2→v5 binary format change breaks all Kryo-serialized state
ValueState<ThirdPartyType> state; // Any type from external libraries using Kryo fallback
```

**Any Type Using Kryo Fallback:**
If Flink cannot handle a type with a built-in or registered serializer, it falls back to Kryo. All Kryo-serialized state from 1.x is incompatible with 2.2. Use `env.getConfig().disableGenericTypes()` during development to detect Kryo fallback.

### Failure Behavior and Detection

**Failure Path:**

1. Managed Service for Apache Flink application update completes successfully (no errors during deployment)
2. Job transitions to RUNNING state
3. State restoration begins during operator initialization
4. Deserialization fails with serializer mismatch
5. Job transitions to FAILED state
6. Restart strategy triggers automatic restart
7. **Infinite restart loop** - same failure repeats until manual intervention

#### Critical: No Sync API Feedback

- Update API returns success before state restoration
- Failures occur during state restoration path (outside async workflow scope)
- State restoration duration varies (seconds to tens of minutes based on state size)
- **Only visible through metrics**: downtime, restart count, uptime

**Error Signatures to Watch For:**

*Kryo Reference Tracking:*

```
com.esotericsoftware.kryo.KryoException: Unable to resolve reference for String with id: 116
Caused by: java.lang.IndexOutOfBoundsException: Index 116 out of bounds for length 1
```

*Kryo CollectionSerializer:*

```
com.esotericsoftware.kryo.KryoException: Unable to find class: value2
Caused by: java.lang.ClassNotFoundException: value2
```

*PojoSerializer Collection Handling:*

```
org.apache.flink.util.StateMigrationException: The new state serializer 
(org.apache.flink.api.java.typeutils.runtime.PojoSerializer@8bf85b5d) must not be 
incompatible with the old state serializer 
(org.apache.flink.api.java.typeutils.runtime.PojoSerializer@3282ee3)
```

### Pre-Migration Assessment

**Identify Incompatible Patterns:**

1. **Search for Kryo registration:**

```java
grep -r "registerTypeWithKryoSerializer" src/
grep -r "registerKryoType" src/
```

1. **Search for collection fields in POJOs:**

```java
// Look for POJOs with List, Map, Set, Collection, Queue, Deque fields
// that are used in state (ValueState, MapState, etc.)
```

1. **Search for generic collection state:**

```java
grep -r "ValueState<List" src/
grep -r "ValueState<Map" src/
grep -r "ValueState<Set" src/
```

1. **Check for Kryo fallback warnings in logs:**

```
"Class ... cannot be used as a POJO type because not all fields are valid POJO fields"
```

### Migration Strategies for Incompatible Apps

#### Strategy 1: Parallel Deployment (Zero Downtime)

- Deploy Flink 2.x application alongside 1.x
- Let 2.x rebuild state from scratch while 1.x continues processing
- Switch traffic once 2.x state is fully built
- Best for: Applications that can tolerate dual processing temporarily

#### Strategy 2: State Processor API (State Transformation)

- Use State Processor API to read 1.x savepoint
- Transform state serialization to 2.x-compatible format (Avro/Protobuf)
- Write new savepoint for 2.x restoration
- Best for: Large state that cannot be rebuilt quickly

#### Strategy 3: Reprocess from Source (Clean Start)

- Take final savepoint from 1.x for audit/rollback
- Deploy 2.x with refactored serialization (Avro/Protobuf)
- Reprocess historical data from Kinesis/Kafka/S3
- Best for: Applications with replayable sources and acceptable rebuild time

#### Strategy 4: Refactor Before Migration (Recommended)

- Refactor 1.x application to use compatible serialization (Avro/Protobuf)
- Deploy refactored 1.x, let it checkpoint with new serialization
- Then migrate to 2.x with state preservation
- Best for: Minimizing risk and ensuring future compatibility

### Best Practices for Migration-Safe Applications

**Use Schema-Based Serialization:**

```java
// RECOMMENDED: Avro with explicit TypeInformation
import org.apache.flink.formats.avro.typeutils.AvroTypeInfo;

ValueStateDescriptor<UserEvent> descriptor = new ValueStateDescriptor<>(
    "user-events",
    AvroTypeInfo.of(UserEvent.class)
);
```

**Avoid Collection Fields in POJOs:**

```java
// INSTEAD OF:
public class UserSession {
    public List<String> events; // BREAKS on migration
}

// USE:
public class UserSession {
    public String eventsJson; // Serialize collections as strings
    // Or use Avro schema with array types
}
```

**Use MapState Instead of Collections:**

```java
// INSTEAD OF:
ValueState<Map<String, Integer>> mapState; // BREAKS

// USE:
MapState<String, Integer> mapState; // Compatible - Flink's built-in state type
```

**Detect Kryo Fallbacks During Development:**

```java
// Add during development to catch serialization issues early
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.getConfig().disableGenericTypes(); // Throws exception if Kryo would be used
```

## MSF Upgrade Workflow

### Understanding Migration Paths

Your upgrade experience depends on your application's compatibility with Flink 2.2:

**Path 1: Compatible binary and state** — Invoke the Upgrade operation. Application transitions RUNNING → UPDATING → RUNNING with full state preservation. Same experience as minor version migrations. Best for stateless applications or those using compatible serialization (Avro, Protobuf, simple POJOs without collections).

**Path 2: Binary incompatibilities** — Upgrade operation fails and surfaces the incompatibility through Operations API and logs. With auto-rollback enabled, the application automatically rolls back within minutes. With auto-rollback disabled, the application remains running without processing data until you manually roll back. Fix the binary issues, then re-attempt for a Path 1 experience.

**Path 3: Incompatible application state** — Upgrade appears to succeed initially, but the application enters restart loops within seconds as state restoration fails. Detect via CloudWatch metrics (`numRestarts` increasing, `runningTime` not increasing). Manually invoke the Rollback operation, then review the State Compatibility section above.

### Upgrade Phases

#### Phase 1: Preparation

1. Update application code for Flink 2.2 compatibility (dependencies, removed APIs, Java 17)
2. Build the new JAR and upload to S3 with a different file name (e.g., `my-app-flink-2.2.jar`)

#### Phase 2: Enable auto-rollback

Check and enable auto-rollback before upgrading:

```bash
# Check auto-rollback status
aws kinesisanalyticsv2 describe-application \
    --application-name MyApplication \
    --query 'ApplicationDetail.ApplicationConfigurationDescription.ApplicationSystemRollbackConfigurationDescription.RollbackEnabled'

# Enable if not already enabled
aws kinesisanalyticsv2 update-application \
    --application-name MyApplication \
    --current-application-version-id <version-id> \
    --application-configuration-update '{
        "ApplicationSystemRollbackConfigurationUpdate": {
            "RollbackEnabledUpdate": true
        }
    }'
```

#### Phase 3: Take snapshot

If automatic snapshots are enabled, you can skip this. Otherwise, take a snapshot before upgrading:

```bash
aws kinesisanalyticsv2 create-application-snapshot \
    --application-name MyApplication \
    --snapshot-name pre-flink-2.2-upgrade

# Wait until READY
aws kinesisanalyticsv2 describe-application-snapshot \
    --application-name MyApplication \
    --snapshot-name pre-flink-2.2-upgrade
```

#### Phase 4: Upgrade application

You can upgrade from RUNNING or READY state using the UpdateApplication API:

```bash
aws kinesisanalyticsv2 update-application \
    --application-name MyApplication \
    --current-application-version-id <version-id> \
    --runtime-environment-update FLINK-2_2 \
    --application-configuration-update '{
        "ApplicationCodeConfigurationUpdate": {
            "CodeContentUpdate": {
                "S3ContentLocationUpdate": {
                    "FileKeyUpdate": "my-app-flink-2.2.jar"
                }
            }
        }
    }'
```

CloudFormation also supports in-place upgrades — update the `RuntimeEnvironment` field and CloudFormation will update in place without deleting and recreating the application (preserving snapshots and history).

#### Phase 5: Monitor upgrade

- Use the Operations API to check upgrade status and surface binary incompatibilities
- If the application is RUNNING but still on the older runtime, auto-rollback kicked in — check Operations API for the failure reason
- Monitor CloudWatch metrics:
  - `numRestarts`: should be zero after upgrade
  - `runningTime`: should be steadily increasing (replaces deprecated `uptime`)
  - `lastCheckpointDuration`: should be similar to pre-upgrade values
  - `numberOfFailedCheckpoints`: should remain at 0

#### Phase 6: Validate (run 24+ hours)

- Verify data flowing through sources and sinks
- Compare output with pre-upgrade baseline
- Monitor latency, throughput, checkpoint duration/size, memory/CPU utilization

### Rollback Procedures

**Automatic rollback:** If auto-rollback is enabled and the upgrade fails during startup, MSF automatically reverts to the previous version.

**Manual rollback** (for applications running but unhealthy):

```bash
aws kinesisanalyticsv2 rollback-application \
    --application-name MyApplication \
    --current-application-version-id <version-id>
```

Rollback restores the previous Flink version, previous JAR, and restarts from the last snapshot taken before the upgrade. You cannot restore a Flink 2.2 snapshot on Flink 1.x.

### Rebuild State (for incompatible state)

If state is incompatible and cannot be migrated, start fresh:

```bash
aws kinesisanalyticsv2 start-application \
    --application-name MyApplication \
    --run-configuration '{
        "ApplicationRestoreConfiguration": {
            "ApplicationRestoreType": "SKIP_RESTORE_FROM_SNAPSHOT"
        }
    }'
```

## Build Process

Set Java 17 before building:

```bash
# Set JAVA_HOME to your Java 17 installation path before building
export JAVA_HOME=<path-to-java-17>
mvn clean package -DskipTests
```

## Removed APIs and Migration Paths

### DataSet API Removal

The entire DataSet API for batch processing has been removed. All batch processing must use the unified DataStream API or Table API/SQL:

```java
// REMOVED - DataSet API
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
DataSet<String> data = env.readTextFile("input.txt");

// USE instead - DataStream API or Table API/SQL
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// Use DataStream API for batch with bounded sources, or Table API/SQL
```

### Scala API Removal

The Flink Scala API has been removed entirely. Scala-specific wrappers and implicit conversions are no longer available. Use Flink's Java API from Scala code instead.

### Source API Migration

`SourceFunction` is completely removed. Migrate to the new Source API:

```java
// REMOVED in 2.x
env.addSource(new MySourceFunction<>());

// USE instead - new Source API
env.fromSource(
    mySource,
    WatermarkStrategy.forMonotonousTimestamps(),
    "source-name"
);
```

### Sink API Migration

`SinkFunction` and `SinkV1` are removed. Migrate to Sink V2:

```java
// REMOVED in 2.x
stream.addSink(new MySinkFunction<>());

// USE instead - Sink V2 API
stream.sinkTo(mySinkV2);
```

### DataStream API Removals

```java
// REMOVED - IterativeStream (feedback loops)
stream.iterate();
stream.iterate(5000);

// REMOVED - TimeCharacteristic (event time is now default)
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);

// REMOVED - keyBy with field positions/names
stream.keyBy(0);           // Use KeySelector instead
stream.keyBy("fieldName"); // Use KeySelector instead

// REMOVED - partitionCustom with field positions
stream.partitionCustom(partitioner, 0);

// REMOVED - legacy file methods
env.readTextFile("path");
stream.writeAsText("path");
stream.writeAsCsv("path");

// REMOVED - legacy window methods
stream.timeWindowAll(Time.seconds(10));
```

### Configuration API Removals

```java
// REMOVED - string-based configuration methods
config.setInteger("key", value);
config.getInteger("key", default);
config.setBoolean("key", value);
// ... all primitive setters/getters with string keys

// REMOVED - Kryo registration methods from StreamExecutionEnvironment
env.registerTypeWithKryoSerializer(MyClass.class, MySerializer.class);
env.addDefaultKryoSerializer(MyClass.class, MySerializer.class);
env.registerType(MyClass.class);

// REMOVED - legacy state backend setter
env.setStateBackend(stateBackend);

// REMOVED - restart strategy methods
env.setRestartStrategy(restartStrategy);
env.getRestartStrategy();
env.setNumberOfExecutionRetries(3);
```

### Configuration File Migration

Legacy `flink-conf.yaml` is no longer supported. Use `config.yaml` with standard YAML format:

```yaml
# config.yaml (new format)
jobmanager:
  rpc:
    address: localhost
    port: 6123
  memory:
    process:
      size: 1600m

taskmanager:
  memory:
    process:
      size: 1728m
  numberOfTaskSlots: 4

parallelism:
  default: 4
```

A migration tool is provided: see Flink documentation for "Migrate from flink-conf.yaml to config.yaml".

### Window Assigner Changes

Window assigners now use `Duration` instead of `Time`:

```java
// REMOVED - Time-based factory methods
TumblingEventTimeWindows.of(Time.seconds(10));
SlidingProcessingTimeWindows.of(Time.minutes(1), Time.seconds(10));
EventTimeSessionWindows.withGap(Time.minutes(5));

// USE instead
TumblingEventTimeWindows.of(Duration.ofSeconds(10));
SlidingProcessingTimeWindows.of(Duration.ofMinutes(1), Duration.ofSeconds(10));
EventTimeSessionWindows.withGap(Duration.ofMinutes(5));
```

### Connector Compatibility

First-party connectors require 2.x compatible versions. Check connector documentation for migration status:

| Connector | Flink 1.20 Version | Flink 2.0+ Version | Notes |
|-----------|--------------------|--------------------|-------|
| Apache Kafka | `flink-connector-kafka` 3.4.0-1.20 | `flink-connector-kafka` 4.0.1-2.0 | Recommended for Flink 2.2 |
| Kinesis Data Streams (source) | `flink-connector-aws-kinesis-streams` 5.1.0-1.20 (or legacy `flink-connector-kinesis` 5.0.0-1.20) | `flink-connector-aws-kinesis-streams` 6.0.0-2.0 | Must be on v5.0+ before upgrade |
| Kinesis Data Streams (sink) | `flink-connector-aws-kinesis-streams` 5.1.0-1.20 | `flink-connector-aws-kinesis-streams` 6.0.0-2.0 | Recommended for Flink 2.2 |
| Amazon Data Firehose | `flink-connector-aws-kinesis-firehose` 5.1.0-1.20 | `flink-connector-aws-kinesis-firehose` 6.0.0-2.0 | Compatible with Flink 2.0 |
| Amazon DynamoDB | `flink-connector-dynamodb` 5.1.0-1.20 | `flink-connector-dynamodb` 6.0.0-2.0 | Compatible with Flink 2.0 |
| Amazon SQS | `flink-connector-sqs` 5.1.0-1.20 | `flink-connector-sqs` 6.0.0-2.0 | Compatible with Flink 2.0 |
| FileSystem (S3, HDFS) | Bundled with Flink | Bundled with Flink | Always available |
| JDBC | `flink-connector-jdbc` 3.3.0-1.20 | Not yet released for 2.x | No Flink 2.x-compatible release |
| OpenSearch | `flink-connector-opensearch` 1.2.0-1.19 | Not yet released for 2.x | No Flink 2.x-compatible release |
| Elasticsearch | Legacy connector only | Not yet released for 2.x | Consider migrating to OpenSearch connector |
| Amazon Managed Service for Prometheus | `flink-connector-prometheus` 1.0.0-1.20 | Not yet released for 2.x | No Flink 2.x-compatible release |

Note: Some connectors were renamed between major versions (e.g., Firehose connector). Always check the MSF connector documentation for exact artifact names.

## New Features in 2.x

### Disaggregated State Management

Flink 2.x introduces ForSt (disaggregated state backend) for cloud-native deployments:

- Leverages distributed file systems as primary storage
- Asynchronous execution model for better performance
- Fast rescaling for large state (hundreds of TB)
- Reduced local disk requirements

### DataStream V2 API (Experimental)

New DataStream API with improved design (not yet production-ready):

- Cleaner separation of concerns
- Better state management primitives
- Improved type safety

### SQL and Table API Enhancements

| Feature | Description |
|---------|-------------|
| VARIANT data type | Native support for semi-structured data (JSON) without repeated string parsing |
| Delta Join | Reduces state for streaming joins by maintaining only latest version per key (requires external infrastructure like Apache Fluss) |
| StreamingMultiJoinOperator | Executes multi-way joins as a single operator, eliminating intermediate materialization |
| ProcessTableFunction (PTF) | Stateful, event-driven logic directly in SQL with per-key state and timers |
| ML_PREDICT function | Call registered ML models on streaming/batch tables from SQL (requires bundling a ModelProvider implementation) |
| Model DDL | Define ML models as first-class catalog objects using `CREATE MODEL` statements |
| Vector Search | SQL API for searching vector databases (requires custom VectorSearchTableSource implementation) |
| C-style escape strings | Supported in SQL |
| QUALIFY clause | Filter window function outputs |

### DataStream API

| Feature | Description |
|---------|-------------|
| FLIP-27 Source API | New unified source interface replacing legacy `SourceFunction` |
| FLIP-143 Sink API | New unified sink interface replacing legacy `SinkFunction` |
| Async Python DataStream | Non-blocking I/O in Python DataStream API using `AsyncFunction` |

### Runtime

- RocksDB upgraded to 8.10.0 with improved I/O performance
- Dedicated serializers for Map, List, Set (replacing Kryo-based serialization)
- Python 3.12 support

## Quick Reference: Breaking Changes

| Category | 1.x | 2.x |
|----------|-----|-----|
| Java version | 8, 11+ | 17+ (21 experimental, not supported in MSF) |
| Python version | 3.8+ | 3.12 (3.8 removed) |
| Scala API | Available | Removed (use Java API from Scala) |
| DataSet API | Available | Removed (use DataStream or Table API) |
| Config file | `flink-conf.yaml` | `config.yaml` (standard YAML) |
| Source API | `SourceFunction` | New Source API only |
| Sink API | `SinkFunction`, `SinkV1` | Sink V2 only |
| Time API | `Time.seconds(n)` | `Duration.ofSeconds(n)` |
| Config API | `config.setInteger("key", val)` | `config.set(ConfigOption, val)` |
| Function open() | `open(Configuration)` | `open(OpenContext)` |
| Logging bridge | `slf4j-log4j12` | `log4j-slf4j-impl` |
| CEP select | Implicit type inference | Explicit `TypeInformation` required |
| Kryo version | 2.x | 5.x (incompatible state format) |
| Collection serializers | Kryo-based | Dedicated serializers (FLINK-34123) |
| TimeCharacteristic | Configurable | Removed (event time default) |
| IterativeStream | Supported | Removed |
| keyBy(int) | Supported | Removed (use KeySelector) |
| Scala case classes state | Kryo v2 serialized | Incompatible (Kryo v5) |
| Java records state | Kryo v2 fallback | Incompatible (Kryo v5) |
| MSF filesystem | Writable | Read-only (except `/tmp`) |
| MSF IMDS | All endpoints | Credential endpoints only |
| CloudFormation upgrade | Delete and recreate | In-place `RuntimeEnvironment` update |

## MSF Behavioral Changes in Flink 2.2

### Metrics Changes

| Change | Details |
|--------|---------|
| `fullRestarts` removed | Use `numRestarts` instead |
| `uptime` deprecated | Use `runningTime` instead |
| `downtime` deprecated | Use `restartingTime`, `cancellingTime`, `failingTime` instead |
| `bytesRequestedPerFetch` removed | Removed in KDS connector v6.0.0 |

### Read-Only Root Filesystem

To improve security, any file write outside of `/tmp` (the default Flink working directory) will fail with `java.io.FileNotFoundException: /{path}/{filename} (Read-only file system)`. This can come from your code directly or from libraries in your dependencies. Override direct filesystem paths to `/tmp/` in your code, and use library configuration overrides to redirect indirect filesystem operations to `/tmp/`.

### Non-Credential IMDS Calls Blocked

Only credential-related IMDS endpoints are allowed (`/latest/meta-data/iam/security-credentials/` and `/latest/dynamic/instance-identity/document`). Applications using other IMDS calls (e.g., `EC2MetadataUtils.getInstanceId()`, `getInstanceType()`, `getLocalHostName()`, `getAvailabilityZone()`) will receive HTTP 4xx errors. Refactor to use environment variables or application configuration instead.

### Programmatic Configuration Handling

MSF Flink 2.2 now throws an exception when you attempt to modify configs not supported by MSF through `env.getConfig().set()` or similar APIs. Supported config changes can still be requested through support tickets.

### Known Issues

**MSF Studio not supported:** Flink 2.2 in MSF does not support Studio (notebook) applications.

**Kinesis EFO resharding bug (FLINK-37648):** Applications using `KinesisStreamsSource` with EFO (SubscribeToShard) may fail when Kinesis streams undergo resharding.

**Kinesis EFO + Sink deadlock (FLINK-34071):** Applications using `KinesisStreamsSource` with EFO together with `KinesisStreamsSink` may experience deadlocks under backpressure, resulting in complete stop of data processing. Recovery requires a force stop and restart.

## Not Supported Features in Managed Service for Apache Flink

The following Flink 2.2 features are not currently supported in Managed Service for Apache Flink as they are still considered experimental in Apache Flink:

- Materialized Tables
- ForSt State Backend (disaggregated state storage)
- Java 21
- Custom metric reporters/telemetry configurations

For details on which features are supported in Managed Service for Apache Flink, refer to  [Apache Flink 2.2 features supported](https://docs.aws.amazon.com/managed-flink/latest/java/flink-2-2.html#flink-2-2-supported-features).

## References

- [Flink 2.0 Release Announcement](https://flink.apache.org/2025/03/24/apache-flink-2.0.0-a-new-era-of-real-time-data-processing/)
- [FLINK-34037](https://issues.apache.org/jira/browse/FLINK-34037) - Serialization configuration changes
- [FLINK-34123](https://issues.apache.org/jira/browse/FLINK-34123) - Collection serializer changes
- [FLIP-398](https://cwiki.apache.org/confluence/display/FLINK/FLIP-398) - Serialization improvements

# Foundation Operations: Quotas, Service Selection, and Architecture

## Overview

Cross-cutting operational knowledge: MSF service quotas, ENI capacity planning for VPC apps, MSF vs EMR Flink decision, and source/sink selection. Use when planning capacity, choosing the right streaming service, or designing a new pipeline.

## CLI and CloudWatch Identifiers

| Item | Value | Common wrong values |
|------|-------|---------------------|
| AWS CLI service / SDK client | `kinesisanalyticsv2` | ❌ `flink`, `msf`, `kinesisanalytics` (v1, deprecated SQL apps only) |
| Service Quotas service-code | `kinesisanalytics` (no `v2`) | ❌ `kinesisanalyticsv2`, `flink`, `msf` |
| IAM action prefix | `kinesisanalytics:` (no `v2`) | ❌ `kinesisanalyticsv2:` |
| CloudWatch namespace | `AWS/KinesisAnalytics` | ❌ `AWS/Flink`, `AWS/ManagedFlink` |
| Trust policy principal | `kinesisanalytics.amazonaws.com` | ❌ `kinesisanalyticsv2.amazonaws.com` |

The CLI/SDK is the *only* identifier that uses the `v2` suffix. Service Quotas, IAM actions, the CloudWatch namespace, and the trust principal all use the legacy `kinesisanalytics` name. Treating the v2 form as the "default" and applying it everywhere is the single most common source of permission failures, empty metric results, missing service-quota lookups, and trust policy errors.

## What Goes In the Execution Role (and What Does NOT)

The MSF execution role is assumed by the **MSF service** to access **your data plane resources** on behalf of the application. It is not used by the application code itself, and it does not call MSF's own control plane. The principle of least privilege follows from that:

| Permission | Required in execution role? | Why |
|------------|------------------------------|-----|
| Source/sink data plane (e.g. `kinesis:GetRecords`, `s3:PutObject`, `kafka:DescribeCluster`) | Yes — scoped to specific stream/bucket/cluster ARNs | The service uses this role to read sources and write sinks |
| `logs:CreateLogStream`, `logs:PutLogEvents`, `logs:DescribeLogStreams` on the configured log group | Yes | The service writes application logs to the configured CloudWatch Logs group |
| EC2 ENI permissions (`ec2:CreateNetworkInterface`, `ec2:DescribeNetworkInterfaces`, `ec2:DeleteNetworkInterface`, `ec2:CreateNetworkInterfacePermission`, `ec2:DescribeVpcs`, `ec2:DescribeSubnets`, `ec2:DescribeSecurityGroups`) | Yes — only for VPC-enabled apps | The service creates ENIs on your behalf when VPC is configured |
| `kinesisanalytics:*` actions | **No** | These are MSF control-plane actions consumed by humans/CI calling the MSF API, not by the service when it runs your application |
| `cloudwatch:PutMetricData` | **No** | MSF publishes the standard metrics (cpuUtilization, downtime, numRecordsIn*, etc.) to the `AWS/KinesisAnalytics` namespace from the service plane, not via the execution role. Adding it does no harm if scoped to the namespace, but it's noise — leave it out for a clean least-privilege role. The exception: if your **application code** explicitly calls `CloudWatchAsyncClient.putMetricData()` to emit custom application metrics, then you do need to grant it (still scope to the custom namespace) |
| `secretsmanager:GetSecretValue` | Only if you use Secrets Manager for connector credentials | Scope to the specific secret ARN |
| `kms:Decrypt` | Only when reading from a KMS-encrypted source/sink or KMS-encrypted secret | Scope to the specific key ARN |

When generating an execution role, default to omitting `kinesisanalytics:*` and `cloudwatch:PutMetricData` unless the user explicitly says they emit custom metrics. The CloudWatch metrics you see in the AWS console come from the service, not from this role.

## Service Quotas

| Quota | Default | Adjustable |
|-------|---------|------------|
| Applications per region | 50 | Yes |
| KPUs per application | 64 (configurable to 250) | Yes |
| Application snapshots per application | 1000 | Yes |
| Parallelism per application | 256 | Yes |

Check current values:

```bash
aws service-quotas list-aws-default-service-quotas \
  --service-code kinesisanalytics --region "$REGION"
```

The Service Quotas service-code is `kinesisanalytics` (legacy name), not `kinesisanalyticsv2`. The KPUs-per-application quota code is `L-3A88E041` (`Apache Flink Kinesis Processing Units (KPUs)`).

Request increases via `request-service-quota-increase` with the quota code from the list output. Increases for KPU and snapshot quotas typically approve within a business day.

## ENI Capacity Planning (VPC Apps)

MSF creates one ENI per allocated KPU, in each subnet, for VPC-enabled applications. Subnet sizing and the regional ENI quota both matter.

```
ENIs_per_subnet = KPUs
required_subnet_IPs = ENIs_per_subnet + 20% headroom for scaling and rolling restarts
```

Example: 16 KPU app 16 ENIs per subnet. Allow 20 available IPs per subnet to account for 20% buffer. A `/28` subnet (11 usable IPs) is too small; use `/27` (27 usable) or larger.

**Regional ENI quota** (`vpc` service code, `Network interfaces per region`) defaults to 5,000. Each VPC-enabled MSF KPU consumes one. Multiple large MSF apps in the same region can pressure this quota — check before deploying. Other services in the VPC (Lambda, ECS, RDS) also consume ENIs from the same quota.

Subnets should span at least 2 AZs for fault tolerance.

## When VPC Is Required

| Source/Sink | VPC Required |
|-------------|--------------|
| Kinesis Data Streams | No (public endpoint) |
| Amazon S3 | No (public endpoint, S3 gateway endpoint optional) |
| DynamoDB | No (public endpoint) |
| Firehose | No |
| Amazon MSK (Kafka) | **Yes** |
| RDS / Aurora | **Yes** |
| ElastiCache | **Yes** |
| OpenSearch in VPC | **Yes** |
| Self-hosted Kafka | **Yes** if private |
| Public Kafka / external API | No, but needs NAT gateway in VPC subnets |

VPC apps without a NAT gateway lose access to public AWS endpoints (CloudWatch, S3 if not using gateway endpoint, Kinesis). Symptoms: silent failure of metric publishing, S3 checkpoint failures, deserialization errors trying to call schema registry.

## MSF vs EMR Flink

| Factor | MSF | EMR Flink |
|--------|-----|-----------|
| Operations | Fully managed, no clusters | Self-managed EC2/EKS clusters |
| Scaling | KPU autoscaling (CPU-only) | Manual cluster scaling |
| Billing | Per KPU-hour | Per EC2-hour (+ EMR surcharge) |
| Flink version | AWS-managed (1.15, 1.18, 1.19, 1.20, 2.2) | Any Flink version, including custom builds |
| Custom connectors | Limited to bundled JARs / fat-JAR upload | Full Flink ecosystem |
| Job isolation | One job per application | Multiple jobs per cluster |
| Startup time | 1–3 min | 5–15 min cluster boot |
| Max parallelism | 256 (quota-adjustable) | Unlimited (cluster size) |
| State backend | Managed RocksDB | Self-managed RocksDB / heap |

**Choose MSF when:** zero infrastructure management, single-job-per-app is acceptable, parallelism fits within KPU limits, fast iteration matters.

**Choose EMR Flink when:** custom Flink connectors not in MSF, multiple Flink jobs sharing infrastructure, specific Flink version or patch control, parallelism exceeds 256, fine-grained CPU/memory ratios needed.

**Cost crossover:** EMR is typically cheaper at large scale (10+ KPU equivalent, 24/7) due to EC2 commitment savings, but the operational overhead (cluster patching, scaling, monitoring) typically erases the savings unless EMR expertise already exists.

## Source Selection: KDS vs MSK vs S3

| Source | Best For | Throughput | VPC | Ordering | Retention |
|--------|---------|-----------|-----|----------|-----------|
| Kinesis Data Streams (KDS) | AWS-native ingestion, < 1 GB/s, Lambda integration | Per-shard (1 MB/s in, 2 MB/s out) | No | Per-shard | 1–365 days |
| Amazon MSK | Kafka ecosystem, complex routing, > 1 GB/s | Per-broker (hundreds MB/s) | Yes | Per-partition | Unlimited (storage-based) |
| Amazon S3 | Batch-to-stream replay, reprocessing | Bulk file scan | No | File ordering | Indefinite |

**KDS pitfalls:** shard count drives parallelism; shard count is hard to change post-creation. Cost scales linearly with shard count, not throughput.

**MSK pitfalls:** broker provisioning takes hours; cross-AZ replication doubles network cost; SASL/IAM auth requires careful security group setup.

**S3 source pitfalls:** no event-time ordering across files unless designed in; file enumeration is the bottleneck for large prefix counts.

For source-side EFO and Kinesis polling tradeoffs, see [kinesis-efo-guide.md](kinesis-efo-guide.md). For MSK setup, see the [iac-and-deployment.md](iac-and-deployment.md) deployment patterns.

## Sink Selection

| Sink | Best For | VPC | Ordering Preserved |
|------|---------|-----|--------------------|
| S3 (Parquet/ORC) | Data lake, batch analytics, Athena | No | File-level only |
| Iceberg | Transactional data lake, schema evolution, time travel | No (catalog-dependent) | Yes (commit-order) |
| Kinesis Data Streams | Real-time downstream consumers | No | Per-shard |
| MSK | Kafka ecosystem | Yes | Per-partition |
| OpenSearch | Search, log analytics, dashboards | Yes (VPC mode) or No (public) | No (eventually consistent) |
| DynamoDB | Low-latency key-value lookups | No | No (last-write-wins per key) |
| RDS / Aurora | Relational writes, joins on results | Yes | Yes (transaction order) |
| Firehose | Managed delivery to S3/Redshift/OpenSearch with batching | No | Within batch |

**S3 small files anti-pattern:** Streaming writers commit on every checkpoint, creating one file per checkpoint per partition. With 60s checkpoints and 8 partitions, a job writes 11,520 files/day. Use Iceberg with compaction, or batch via Firehose, or increase checkpoint interval. See [iceberg-tuning-and-operations.md](iceberg-tuning-and-operations.md).

**RDS/Aurora sink anti-pattern:** Per-record JDBC writes are 10–100× slower than batched writes. Use the JDBC sink with batching enabled. RDS connection pool limits (typically 5,000) cap effective parallelism.

## Architecture Patterns

| Pattern | Use Case | VPC |
|---------|---------|-----|
| KDS → MSF → S3/Iceberg | Data lake ingestion | No |
| MSK → MSF → MSK | Stream-to-stream enrichment | Yes |
| KDS → MSF → DynamoDB | Real-time aggregation serving | No |
| KDS → MSF → OpenSearch | Real-time search index | Optional |
| CDC source → MSF → Iceberg | Database replication to lake | Yes (DB side) |
| MSK → MSF → S3 + KDS | Fan-out to lake and downstream consumers | Yes (MSK side) |

## Co-location and Cross-AZ Cost

For private connections (MSK, RDS, OpenSearch in VPC), MSF KPUs must be in the same VPC or peered VPC as the source/sink. Cross-AZ data transfer between MSF and MSK is **always billed** at $0.01/GB each direction — at 100 MB/s sustained that is $26K/year. Place MSF subnets in the same AZs as MSK brokers and use rack-aware producers/consumers where supported.

## Common Mistakes

| Mistake | Impact |
|---------|--------|
| Using `kinesisanalytics` for the CLI / SDK control plane | Returns v1 SQL apps only, not Flink — use `kinesisanalyticsv2` for the CLI/SDK |
| `kinesisanalyticsv2` for the Service Quotas service-code | `NoSuchResourceException` — Service Quotas uses the legacy `kinesisanalytics` code |
| `kinesisanalyticsv2:` IAM actions | All API calls denied — IAM uses `kinesisanalytics:` prefix |
| `AWS/Flink` namespace in CloudWatch | Empty metric results |
| `kinesisanalyticsv2.amazonaws.com` trust principal | MSF cannot AssumeRole — trust principal is `kinesisanalytics.amazonaws.com` |
| Hardcoding region lists for cross-region discovery | Disabled regions cause `AccessDeniedException` — use `describe-regions` |
| Sizing subnets without ENI headroom | Application stuck STARTING; ENIs cannot be created |
| Choosing EMR for a single-job pipeline | Operational overhead exceeds infrastructure savings |
| Mismatched MSF and source AZs | Persistent cross-AZ data transfer cost |

## References

- [MSF Quotas](https://docs.aws.amazon.com/managed-flink/latest/java/limits.html)
- [MSF VPC Configuration](https://docs.aws.amazon.com/managed-flink/latest/java/vpc.html)
- [MSF Pricing](https://aws.amazon.com/managed-service-apache-flink/pricing/)
- [Cross-AZ Data Transfer Pricing](https://aws.amazon.com/ec2/pricing/on-demand/)

# Infrastructure as Code and Deployment Guide

## Overview

This guide covers IaC patterns and deployment automation for Amazon Managed Service for Apache Flink applications. Use it when generating CloudFormation, CDK, Terraform, or deployment scripts for MSF applications. The patterns here address common deployment ordering issues, IAM configuration, and resource dependencies that are specific to MSF.

## CRITICAL: MSF Application JAR Dependency

**The most common IaC failure for MSF is a deployment ordering problem: the MSF application resource requires the application JAR to exist in S3 at creation time.**

When CloudFormation (or any IaC tool) creates a `AWS::KinesisAnalyticsV2::Application` resource with `ApplicationConfiguration.ApplicationCodeConfiguration` pointing to an S3 bucket and key, the MSF service validates that the JAR exists during resource creation. If the JAR is not yet uploaded, the stack fails with:

```
Resource handler returned message: "Please check the role provided or validity of S3 location
you provided. We are unable to get the specified fileKey: <key> in the specified bucket: <bucket>"
```

### Solution: Two-Phase Deployment

**Always structure MSF deployments in two phases:**

1. **Phase 1 — Infrastructure**: Deploy all supporting resources (S3 buckets, Kinesis streams, IAM roles, CloudWatch log groups, VPC resources). This phase does NOT include the MSF application itself.
2. **JAR Upload**: Build the application JAR and upload it to the S3 bucket created in Phase 1.
3. **Phase 2 — Application**: Deploy the MSF application resource, referencing the JAR that now exists in S3.

This applies to all IaC tools: CloudFormation, CDK, Terraform, SAM, etc.

### CloudFormation: Two-Stack Pattern

Split the deployment into two CloudFormation stacks:

**Stack 1 — Infrastructure (`cfn-infra.yaml`)**:

- S3 bucket for JAR staging
- S3 bucket for application output (if applicable)
- Kinesis streams or Kafka/MSK resources
- IAM execution role for the MSF application
- CloudWatch log group and log stream
- VPC, subnets, security groups (if VPC deployment)
- Exports: bucket names, stream ARNs, role ARN, log group/stream ARNs

**Stack 2 — Application (`cfn-app.yaml`)**:

- `AWS::KinesisAnalyticsV2::Application` resource
- `AWS::KinesisAnalyticsV2::ApplicationCloudWatchLoggingOption` (if not inline)
- Imports: references from Stack 1 via `Fn::ImportValue` or parameters

**Deploy script ordering:**

```bash
# 1. Deploy infrastructure
aws cloudformation deploy --template-file cfn-infra.yaml --stack-name my-app-infra ...

# 2. Build and upload JAR
mvn clean package -q
aws s3 cp target/my-app.jar s3://${JAR_BUCKET}/${JAR_KEY}

# 3. Deploy application (JAR now exists in S3)
aws cloudformation deploy --template-file cfn-app.yaml --stack-name my-app ...
```

### CDK: Deployment Ordering with Dependencies

In CDK, use separate stacks or ensure the JAR upload happens before the MSF application construct is created. CDK does not natively upload JARs during synthesis — you need a custom resource or a deploy script wrapper.

**Option A — Two CDK stacks with a script wrapper:**

```typescript
// InfraStack: buckets, streams, IAM, logs
// AppStack: MSF application (depends on InfraStack)
// Deploy script uploads JAR between the two stack deployments
```

**Option B — CDK `BucketDeployment` construct:**

```typescript
import * as s3deploy from 'aws-cdk-lib/aws-s3-deployment';

// Upload JAR to S3 as part of the CDK deployment
const jarDeployment = new s3deploy.BucketDeployment(this, 'JarDeployment', {
  sources: [s3deploy.Source.asset('./target')],
  destinationBucket: jarBucket,
  destinationKeyPrefix: 'jars/',
});

// Ensure MSF application is created AFTER the JAR is uploaded
flinkApp.node.addDependency(jarDeployment);
```

### Terraform: `depends_on` for Upload Ordering

In Terraform, use `aws_s3_object` to upload the JAR and add an explicit `depends_on` to the MSF application resource:

```hcl
resource "aws_s3_object" "app_jar" {
  bucket = aws_s3_bucket.jar_bucket.id
  key    = "jars/my-app.jar"
  source = "target/my-app.jar"
  etag   = filemd5("target/my-app.jar")
}

resource "aws_kinesisanalyticsv2_application" "flink_app" {
  name                   = "my-flink-app"
  runtime_environment    = "FLINK-2_2"  # Default for new apps. Use FLINK-1_20 only when migrating an existing 1.20 app and state compatibility forbids the upgrade.
  service_execution_role = aws_iam_role.flink_role.arn

  application_configuration {
    application_code_configuration {
      code_content {
        s3_content_location {
          bucket_arn = aws_s3_bucket.jar_bucket.arn
          file_key   = aws_s3_object.app_jar.key
        }
      }
      code_content_type = "ZIPFILE"
    }
    # ... other configuration
  }

  depends_on = [aws_s3_object.app_jar]
}
```

## IAM Role Configuration

The MSF application's IAM execution role needs permissions for all AWS resources the application accesses. A common mistake is missing permissions, which causes the application to fail at runtime rather than at deployment.

### Minimum Required Permissions

Every MSF application needs at minimum:

```yaml
# CloudWatch Logs (required for application logging)
# Scope to the application's log group, not log-group:* — that grants logs
# permissions across every group in the account.
- Effect: Allow
  Action:
    - logs:PutLogEvents
    - logs:DescribeLogStreams
  Resource:
    - !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/kinesis-analytics/${ApplicationName}"
    - !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/kinesis-analytics/${ApplicationName}:log-stream:*"

# DescribeLogGroups does not support resource-level permissions
- Effect: Allow
  Action:
    - logs:DescribeLogGroups
  Resource: "*"

# S3 JAR bucket (required to read the application JAR), and KMS permissions if encrypted by a CMK
- Effect: Allow
  Action:
    - s3:GetObject
    - s3:GetObjectVersion
  Resource:
    - !Sub "${JarBucket.Arn}/*"
```

### Common Source/Sink Permissions

**Kinesis Data Streams (source)**:

```yaml
- Effect: Allow
  Action:
    - kinesis:DescribeStream
    - kinesis:GetShardIterator
    - kinesis:GetRecords
    - kinesis:ListShards
    - kinesis:DescribeStreamSummary
    - kinesis:DescribeStreamConsumer
    - kinesis:SubscribeToShard        # Required for EFO
    - kinesis:RegisterStreamConsumer   # Required for EFO
    - kinesis:DeregisterStreamConsumer # Required for EFO
  Resource:
    - !GetAtt KinesisStream.Arn
    - !Sub "${KinesisStream.Arn}/consumer/*"  # Required for EFO
```

**Kinesis Data Streams (sink)**:

```yaml
- Effect: Allow
  Action:
    - kinesis:PutRecord
    - kinesis:PutRecords
    - kinesis:DescribeStream
    - kinesis:DescribeStreamSummary
  Resource:
    - !GetAtt OutputKinesisStream.Arn
```

**S3 (sink)**:

```yaml
- Effect: Allow
  Action:
    - s3:PutObject
    - s3:GetObject
    - s3:ListBucket
    - s3:DeleteObject
    - s3:GetBucketLocation
    - s3:AbortMultipartUpload
    - s3:ListMultipartUploadParts
  Resource:
    - !GetAtt OutputBucket.Arn
    - !Sub "${OutputBucket.Arn}/*"
```

**Kafka/MSK (source or sink)**:

```yaml
# VPC access for MSK. The describe* and *NetworkInterface* actions don't
# accept ARN-scoped resources, so the resource has to be "*" — but you can
# (and should) constrain them with condition keys to the specific VPC/region.
# Example: ec2:Vpc on the network-interface actions, aws:RequestedRegion on
# the describe actions. See guideline 10 (condition keys) below.
- Effect: Allow
  Action:
    - ec2:DescribeVpcs
    - ec2:DescribeSubnets
    - ec2:DescribeSecurityGroups
    - ec2:DescribeDhcpOptions
    - ec2:CreateNetworkInterface
    - ec2:CreateNetworkInterfacePermission
    - ec2:DescribeNetworkInterfaces
    - ec2:DeleteNetworkInterface
  Resource: "*"
  Condition:
    StringEquals:
      aws:RequestedRegion: !Ref AWS::Region
    # For CreateNetworkInterface / CreateNetworkInterfacePermission you can
    # additionally constrain to the application's VPC:
    # ArnEquals:
    #   ec2:Vpc: !Sub "arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:vpc/${VpcId}"
```

### IAM Anti-Patterns

- **Do not use `*` for resource ARNs in production.** Scope permissions to the specific streams, buckets, and log groups the application uses.
- **Do not grant `s3:*` or `kinesis:*`.** Use the minimum set of actions listed above.
- **Do not forget the consumer sub-resource ARN for EFO.** `SubscribeToShard` requires permissions on `stream/*/consumer/*`, not just the stream ARN.

## MSF Application Resource Configuration

### CloudFormation `AWS::KinesisAnalyticsV2::Application`

Key configuration sections and their correct usage:

```yaml
FlinkApplication:
  Type: AWS::KinesisAnalyticsV2::Application
  Properties:
    ApplicationName: !Ref ApplicationName
    RuntimeEnvironment: !Ref FlinkRuntimeEnvironment  # FLINK-2_2 (default for new apps). FLINK-1_20 only for in-place upgrades of existing 1.20 apps.
    ServiceExecutionRole: !GetAtt FlinkRole.Arn
    ApplicationConfiguration:
      # JAR location — JAR must exist before this resource is created
      ApplicationCodeConfiguration:
        CodeContent:
          S3ContentLocation:
            BucketARN: !GetAtt JarBucket.Arn
            FileKey: !Ref JarS3Key
        CodeContentType: ZIPFILE
      # Runtime properties — equivalent to MSF console "Runtime properties"
      EnvironmentProperties:
        PropertyGroups:
          - PropertyGroupId: "kinesis.source"
            PropertyMap:
              stream.arn: !GetAtt InputStream.Arn
              aws.region: !Ref AWS::Region
          - PropertyGroupId: "s3.sink"
            PropertyMap:
              bucket.name: !Ref OutputBucket
              path.prefix: "output/"
      # Parallelism and scaling
      FlinkApplicationConfiguration:
        ParallelismConfiguration:
          ConfigurationType: CUSTOM
          Parallelism: !Ref Parallelism          # Total parallelism (= KPU count × par/KPU)
          ParallelismPerKPU: !Ref ParallelismPerKPU
          AutoScalingEnabled: true
        CheckpointConfiguration:
          ConfigurationType: CUSTOM
          CheckpointingEnabled: true
          CheckpointInterval: 60000
          MinPauseBetweenCheckpoints: 5000
        MonitoringConfiguration:
          ConfigurationType: CUSTOM
          LogLevel: INFO
          MetricsLevel: APPLICATION
```

### Key Configuration Notes

- **`RuntimeEnvironment`**: For new applications, use `FLINK-2_2` (production-recommended default). Use `FLINK-1_20` only when migrating an existing 1.20 application and state compatibility prevents an in-place upgrade (see [flink-2x-migration.md](flink-2x-migration.md) for state-break patterns). The valid enum values come from the `kinesisanalyticsv2` API; the version-segment format mirrors the underlying Flink minor version with an underscore separator (`FLINK-<major>_<minor>`). For the full list of accepted values, see the [`kinesisanalyticsv2 create-application` CLI reference](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/kinesisanalyticsv2/create-application.html), and for migration steps see [flink-2x-migration.md](flink-2x-migration.md).
- **`CodeContentType`**: Always `ZIPFILE` for JAR files (this is the correct value despite the name).
- **`ConfigurationType`**: Set to `CUSTOM` to override defaults. If set to `DEFAULT`, the service ignores your parallelism/checkpoint settings.
- **`Parallelism`**: This is the total parallelism, which equals KPU count × ParallelismPerKPU. For example, 8 KPUs with ParallelismPerKPU=1 means Parallelism=8.
- **`AutoScalingEnabled`**: Set to `true` for production workloads. See [Resource Optimization](resource-optimization.md) for auto-scaling behavior details.
- **`MetricsLevel`**: Use `APPLICATION` for production. `OPERATOR`, `TASK`, and `PARALLELISM` levels increase CloudWatch metric cardinality and cost significantly.

## Deployment Script Patterns

### Build, Deploy, and Start Pattern

A complete deployment script should handle: build → infrastructure deploy → JAR upload → app deploy → code update → start.

**Key considerations:**

- Always build the JAR first and verify it exists before uploading.
- Use `aws cloudformation deploy` (or equivalent) with `--no-fail-on-empty-changeset` to make scripts idempotent.
- After updating the JAR in S3, call `UpdateApplication` with the new S3 object version to point the running application at the new code.
- Starting the application is a separate API call (`StartApplication`) — CloudFormation creates the application in a stopped state.

### Updating a Running Application's Code

To deploy new code to an existing MSF application:

```bash
# 1. Upload new JAR
aws s3 cp target/my-app.jar s3://${JAR_BUCKET}/${JAR_KEY}

# 2. Get current application version
CURRENT_VERSION=$(aws kinesisanalyticsv2 describe-application \
  --application-name my-app \
  --query 'ApplicationDetail.ApplicationVersionId' \
  --output text)

# 3. Update application code reference
aws kinesisanalyticsv2 update-application \
  --application-name my-app \
  --current-application-version-id ${CURRENT_VERSION} \
  --application-configuration-update '{
    "ApplicationCodeConfigurationUpdate": {
      "CodeContentUpdate": {
        "S3ContentLocationUpdate": {
          "BucketARNUpdate": "arn:aws:s3:::'"${JAR_BUCKET}"'",
          "FileKeyUpdate": "'"${JAR_KEY}"'"
        }
      },
      "CodeContentTypeUpdate": "ZIPFILE"
    }
  }'

# 4. Pick up the new code based on current application state.
#    From RUNNING, update-application auto-restarts the app (UPDATING → RUNNING,
#    typically 10–30s downtime depending on state size) — no explicit stop/start needed.
#    From READY (stopped), update-application stays in READY — start the app explicitly
#    to pick up the new code.
#    See application-lifecycle.md for the full state-transition table.
STATUS=$(aws kinesisanalyticsv2 describe-application \
  --application-name my-app \
  --query 'ApplicationDetail.ApplicationStatus' --output text)

if [ "$STATUS" = "READY" ]; then
  aws kinesisanalyticsv2 start-application --application-name my-app \
    --run-configuration '{
      "ApplicationRestoreConfiguration": {
        "ApplicationRestoreType": "RESTORE_FROM_LATEST_SNAPSHOT"
      }
    }'
fi
# Otherwise (RUNNING/UPDATING), poll describe-application until ApplicationStatus
# returns to RUNNING to confirm the auto-restart completed.
```

For guidance on troubleshooting errors after a Flink job upgrade, see [first-fault-isolation.md](first-fault-isolation.md).

Avoid an explicit `stop-application` → `start-application` cycle for code updates on a
RUNNING app. That pattern incurs a full graceful-stop drain plus cold start instead
of the ~10–30s in-place restart that `update-application` performs, and it
contradicts the lifecycle guidance in [application-lifecycle.md](application-lifecycle.md).

### Teardown

When deleting MSF resources:

1. Stop the application first (`StopApplication` API or `Force=true` if stuck).
2. Delete the application stack (MSF application resource).
3. Delete the infrastructure stack (buckets, streams, etc.).
4. S3 buckets with objects require emptying before CloudFormation can delete them — use a custom resource or script.

## CloudFormation vs CDK vs Terraform Comparison for MSF

| Aspect | CloudFormation | CDK | Terraform |
|--------|---------------|-----|-----------|
| JAR upload handling | Manual (script between stack deploys) | `BucketDeployment` construct or manual | `aws_s3_object` resource with `depends_on` |
| Two-phase deployment | Two separate templates | Two stacks or dependency ordering | `depends_on` between resources |
| Application properties | Inline YAML `PropertyGroups` | Typed constructs | HCL `environment_properties` block |
| Drift detection | Supported | Via CloudFormation | Via `terraform plan` |
| Rollback | Automatic on stack failure | Via CloudFormation | Manual `terraform apply` with previous state |

## Common IaC Mistakes to Avoid

1. **Single-stack MSF deployment without pre-uploaded JAR** — The MSF application resource will fail if the JAR doesn't exist in S3. Always use two-phase deployment.
2. **Missing IAM permissions** — The application will start but fail at runtime. Test with the minimum permission set listed above.
3. **Using `ConfigurationType: DEFAULT` with custom values** — The service ignores your parallelism and checkpoint settings. Always use `CUSTOM`.
4. **Hardcoding stream names instead of ARNs** — Use ARNs for cross-account and cross-region compatibility.
5. **Forgetting CloudWatch log permissions** — The application runs but produces no logs, making debugging impossible.
6. **Not setting `CAPABILITY_NAMED_IAM`** — CloudFormation stacks with IAM roles require this capability flag.
7. **S3 bucket cleanup on delete** — CloudFormation cannot delete non-empty S3 buckets. Add a custom resource or use `DeletionPolicy: Retain` and clean up manually.

## References

- See [Best Practices](best-practices.md) for application code patterns and configuration separation
- See [Resource Optimization](resource-optimization.md) for KPU sizing and parallelism configuration
- See [Monitoring and Metrics](monitoring-and-metrics.md) for CloudWatch alarm setup
- See [Logging Configuration](logging-configuration.md) for CloudWatch Logs setup
- See [Kinesis Connector Guide](kinesis-connector-guide.md) for EFO IAM permissions

# Apache Iceberg Integration with Flink on Managed Service for Apache Flink

## Overview

This guide covers building Apache Iceberg applications with Apache Flink on Amazon Managed Service for Apache Flink (MSF): table format selection, write APIs (append, upsert, dynamic, multi-table), distribution modes, read patterns, partitioning strategy, and DDL.

**Mandatory companion file — load before answering catalog or maintenance questions:** [iceberg-tuning-and-operations.md](iceberg-tuning-and-operations.md).

If the user asks about these topics, You MUST also load iceberg-tuning-and-operations.md:

* AWS Glue Catalog vs S3 Tables (which catalog to pick)
* Iceberg table maintenance (compaction, snapshot expiration, orphan cleanup)
* Small files problem
* Flink TableMaintenance API, JDBC locks, post-commit maintenance
* Glue auto-compaction, S3 Tables managed compaction
* Iceberg + Flink dependencies / Maven setup
* Iceberg anti-patterns and monitoring

This file (iceberg-connector-guide.md) does NOT contain the catalog decision matrix or maintenance approaches — those live exclusively in iceberg-tuning-and-operations.md. Answering a catalog or maintenance question from this file alone WILL miss required content.

Iceberg version guidance: Use Iceberg 1.10+ for Flink 1.20, which includes IcebergSink (SinkV2), the TableMaintenance streaming API, delete vectors, and the Dynamic Iceberg Sink. For Flink 2.2, use the corresponding Iceberg release that supports the `iceberg-flink-runtime-2.0` artifact.

## Table Format Version Selection

Iceberg supports three format versions. Choose based on your write pattern:

| Format Version | Use Case | Key Features |
|---|---|---|
| v1 | Append-only workloads | Basic table format, no row-level deletes |
| v2 | Upsert/CDC workloads | Equality deletes, position deletes, row-level operations |
| v3 | Upsert with optimized reads | Delete vectors (more efficient than equality deletes for read performance) |

Set the format version when creating the table:

```sql
-- SQL DDL
CREATE TABLE my_table (
    id BIGINT,
    data STRING,
    PRIMARY KEY (id) NOT ENFORCED
) WITH (
    'format-version' = '2',
    'write.upsert.enabled' = 'true'
);
```

```java
// DataStream API - table creation via Catalog
Map<String, String> tableProperties = new HashMap<>();
tableProperties.put("format-version", "2");
tableProperties.put("write.upsert.enabled", "true");
tableProperties.put("write.delete.mode", "merge-on-read");

catalog.createTable(tableId, schema, partitionSpec, tableProperties);
```

**Guidance:**

* Use v2 for any table that needs upserts, updates, or deletes
* v1 is sufficient for pure append workloads (event logs, clickstreams)
* v3 adds delete vectors which improve read performance for upsert tables, but check query engine compatibility

## Write Patterns

### Choosing a Write API

Iceberg provides two DataStream sink implementations and a SQL path:

| API | Class | When to Use |
|---|---|---|
| IcebergSink (SinkV2) | `IcebergSink` | New applications. Required for table maintenance topology. Supports upsert, branch writes, metrics. |
| FlinkSink (legacy) | `FlinkSink` | Existing applications not yet migrated. Still the default in SQL path unless opted in. |
| SQL INSERT INTO | Table API | SQL-first teams, multi-table routing with StatementSet, simpler pipelines. |
| DynamicIcebergSink | `DynamicIcebergSink` | Dynamic table routing, schema evolution, writing to multiple tables from one stream. |

To use IcebergSink (SinkV2) via SQL, set:

```sql
SET 'table.exec.iceberg.use-v2-sink' = 'true';
```

**Important difference:** IcebergSink uses `uidSuffix` for operator UIDs, while FlinkSink uses `uidPrefix`. When migrating, this affects state compatibility.

### Append Mode (DataStream)

```java
IcebergSink.forRowData(rowDataStream)
    .tableLoader(tableLoader)
    .set("write.format.default", "parquet")
    .set("write.target-file-size-bytes", "134217728")  // 128 MB
    .append();
```

### Upsert Mode (DataStream)

Upsert requires v2+ table format and equality field columns. Partition columns MUST be included in equality fields when using HASH distribution with partitioned tables.

```java
IcebergSink.forRowData(rowDataStream)
    .tableLoader(tableLoader)
    .upsert(true)
    .equalityFieldColumns(Arrays.asList("event_id", "event_date", "region"))
    .set("write.delete.mode", "merge-on-read")
    .set("write.update.mode", "merge-on-read")
    .set("write.merge.mode", "merge-on-read")
    .set("write.format.default", "parquet")
    .set("write.target-file-size-bytes", "134217728")
    .distributionMode(DistributionMode.HASH)
    .append();
```

**Critical upsert rules:**

* Table must use format-version 2 or 3
* Primary key / equality fields must be defined
* Partition columns must be included in equality fields for HASH distribution
* OVERWRITE and UPSERT are mutually exclusive
* Upsert generates equality delete files which accumulate and degrade read performance — compaction is essential
* **HASH distribution is required for correctness with upsert.** Without it (distribution mode NONE), Flink uses rebalance to distribute records across writer tasks. If multiple updates to the same key land on different writer tasks within the same checkpoint, the delete file written by one task cannot find the insert written by another, causing duplicate rows. HASH distribution ensures all records for the same equality fields go to the same writer task. This is a correctness requirement, not just a performance optimization.

### Upsert Mode (SQL)

```sql
CREATE TABLE orders (
    event_id STRING,
    event_date DATE,
    region STRING,
    amount DECIMAL(18, 2),
    PRIMARY KEY (event_id, event_date, region) NOT ENFORCED
) PARTITIONED BY (event_date, region)
WITH (
    'format-version' = '2',
    'write.upsert.enabled' = 'true',
    'write.delete.mode' = 'merge-on-read',
    'write.target-file-size-bytes' = '134217728'
);

-- Upsert via SQL hint (overrides table property per-query)
INSERT INTO orders /*+ OPTIONS('upsert-enabled'='true') */
SELECT * FROM kinesis_source WHERE event_type = 'ORDER';
```

### Multi-Table Routing with StatementSet (SQL)

StatementSet reads the source once and routes to multiple tables efficiently:

```java
StatementSet statementSet = tableEnv.createStatementSet();

statementSet.addInsertSql("INSERT INTO orders SELECT ... FROM kinesis_source WHERE event_type = 'ORDER'");
statementSet.addInsertSql("INSERT INTO users SELECT ... FROM kinesis_source WHERE event_type = 'USER'");
statementSet.addInsertSql("INSERT INTO clicks SELECT ... FROM kinesis_source WHERE event_type = 'CLICK'");

statementSet.execute();  // Single source read, three sinks
```

**Cross-table consistency warning:** Iceberg has no atomic multi-table commit. When a Flink job writes to multiple tables, each table commits independently at checkpoint boundaries. The commits happen sequentially, not atomically. If the job fails between committing table A and table B, downstream queries joining A and B see inconsistent state. Mitigation: each Iceberg snapshot records the Flink checkpoint ID in its summary (`flink.job-id` property). Downstream consumers can query snapshots by checkpoint ID across tables to get a consistent view, but this requires custom read-side logic.

### Dynamic Sink for Schema-Agnostic Routing

The DynamicIcebergSink routes records to tables dynamically, creating tables and evolving schemas automatically:

```java
DynamicIcebergSink.forInput(jsonStream)
    .generator(new MyRoutingGenerator())
    .catalogLoader(catalogLoader)
    .immediateTableUpdate(true)
    .cacheMaxSize(100)
    .cacheRefreshMs(60000)
    .set("write.format.default", "parquet")
    .set("format-version", "2")
    .set("write.target-file-size-bytes", "134217728")
    .append();
```

**Dynamic Sink supports:** adding new columns, widening column types, making required columns optional. It does NOT support dropping or renaming columns.

### Branch Writes

Write to a branch for staging data before merging to main:

```java
IcebergSink.forRowData(rowDataStream)
    .tableLoader(tableLoader)
    .toBranch("staging")
    .append();
```

### Distribution Modes

| Mode | When to Use | Trade-offs |
|---|---|---|
| NONE | Append-only, no partitioning | No shuffle overhead, but may create many small files per partition |
| HASH | Upsert with partitioned tables | Shuffles by partition key or equality fields. Limited by key cardinality. |
| RANGE (experimental) | Skewed data, high-cardinality partitions | Handles skew well, collects traffic statistics. Higher CPU overhead. |

HASH distribution limitation: writer parallelism is capped by the cardinality of the hash key. If you have 10 distinct partition values, only 10 writer tasks receive data regardless of total parallelism.

RANGE distribution handles skewed data (e.g., recent partitions have more traffic than old ones) and can cluster data on non-partition columns when a SortOrder is defined.

## Write Mode Trade-offs: Copy-on-Write vs Merge-on-Read

| Aspect | Copy-on-Write (CoW) | Merge-on-Read (MoR) |
|---|---|---|
| Write cost | High (rewrites entire data files) | Low (writes small delete files) |
| Read cost | Low (no merge logic at query time) | Higher (must merge delete files with data files) |
| Best for | Read-heavy analytical workloads | Write-heavy streaming/CDC workloads |
| Compaction need | Lower | Higher (delete files accumulate) |

**For streaming workloads on MSF, use Merge-on-Read.** It keeps write latency low and checkpoint times fast. Pair it with regular compaction to control read amplification.

### Equality Delete Considerations

Equality deletes are the only viable option for streaming upsert/CDC workloads (the writer doesn't know the physical file location of the row to delete). Key considerations:

* Delete files accumulate with every checkpoint that includes updates/deletes
* Query engines must merge delete files with data files at read time (read amplification)
* Some query engines have limited equality delete support (check your downstream consumers)
* Regular compaction is essential to merge delete files into data files and restore read performance
* Monitor the `equality_delete_file_count` and `equality_delete_record_count` in the `$partitions` metadata table

## Read Patterns

### Streaming Read (DataStream API)

Streaming reads discover new snapshots at a configurable interval. **Streaming reads only work for append-only tables** — tables with upserts (equality deletes) are NOT supported for streaming reads.

```java
IcebergSource<RowData> source = IcebergSource.forRowData()
    .tableLoader(tableLoader)
    .streaming(true)
    .streamingStartingStrategy(StreamingStartingStrategy.INCREMENTAL_FROM_LATEST_SNAPSHOT)
    .monitorInterval(Duration.ofSeconds(60))
    .build();

DataStream<RowData> stream = env.fromSource(
    source,
    WatermarkStrategy.<RowData>forBoundedOutOfOrderness(Duration.ofSeconds(30))
        .withIdleness(Duration.ofMinutes(1)),
    "Iceberg Source",
    TypeInformation.of(RowData.class)
);
```

**Starting strategies:**

* `INCREMENTAL_FROM_LATEST_SNAPSHOT` — Start from latest snapshot (inclusive), discover new appends
* `INCREMENTAL_FROM_EARLIEST_SNAPSHOT` — Start from earliest snapshot (inclusive)
* `TABLE_SCAN_THEN_INCREMENTAL` — Full table scan first, then switch to incremental
* `INCREMENTAL_FROM_SNAPSHOT_ID` — Start from a specific snapshot ID (inclusive)
* `INCREMENTAL_FROM_SNAPSHOT_TIMESTAMP` — Start from a specific timestamp (inclusive)

### Streaming Read (SQL)

```sql
SET table.dynamic-table-options.enabled = true;

-- Read incrementally from latest snapshot
SELECT * FROM my_table /*+ OPTIONS('streaming'='true', 'monitor-interval'='60s') */;

-- Read from a specific snapshot
SELECT * FROM my_table /*+ OPTIONS('streaming'='true', 'monitor-interval'='60s', 'start-snapshot-id'='12345') */;
```

### Batch Read with Time Travel (SQL)

```sql
SET execution.runtime-mode = batch;

-- Read current snapshot
SELECT * FROM my_table;

-- Read from a specific snapshot
SELECT * FROM my_table /*+ OPTIONS('snapshot-id'='12345') */;

-- Read from a specific timestamp
SELECT * FROM my_table /*+ OPTIONS('as-of-timestamp'='1672531200000') */;

-- Read from a branch or tag
SELECT * FROM my_table /*+ OPTIONS('branch'='staging') */;
SELECT * FROM my_table /*+ OPTIONS('tag'='v1.0') */;
```

### Watermark Generation from Iceberg Column Statistics

IcebergSource can generate watermarks from file-level column statistics, useful for windowed processing:

```java
IcebergSource.forRowData()
    .tableLoader(tableLoader)
    .watermarkColumn("event_time")  // timestamp, timestamptz, or long column
    .build();
```

When using watermark columns, set `read.split.open-file-cost` to a large value to prevent combining small files into a single split, which would increase out-of-orderness.

### HybridSource: Bootstrap from Iceberg, Then Stream from Kinesis

The FLIP-150 HybridSource pattern reads historical data from Iceberg (bounded), then seamlessly switches to real-time Kinesis (unbounded):

```java
// Source 1: Iceberg (bounded) - reads all historical data
IcebergSource<RowData> icebergSource = IcebergSource.forRowData()
    .tableLoader(tableLoader)
    .streaming(false)
    .build();

// Source 2: Kinesis (unbounded) - real-time streaming
KinesisStreamsSource<RowData> kinesisSource = KinesisStreamsSource.<RowData>builder()
    .setStreamArn(streamArn)
    .setDeserializationSchema(new JsonToRowDataDeserializer(schema))
    .setSourceConfig(sourceConfig)
    .build();

// Automatic switchover when Iceberg source completes
HybridSource<RowData> hybridSource = HybridSource
    .builder(icebergSource)
    .addSource(kinesisSource)
    .build();
```

Use cases: backfilling new streaming applications, recovering from extended downtime, migrating from batch to streaming.

### Inspecting Tables with Metadata Tables

Query Iceberg metadata tables for operational visibility:

```sql
-- View snapshots (check for flink.job-id in summary)
SELECT snapshot_id, committed_at, operation, summary FROM db.my_table$snapshots;

-- View current data files (detect small files)
SELECT file_path, record_count, file_size_in_bytes FROM db.my_table$files;

-- View partitions (check delete file accumulation)
SELECT partition, file_count, record_count,
       equality_delete_file_count, equality_delete_record_count
FROM db.my_table$partitions;

-- View manifests
SELECT path, added_data_files_count, deleted_data_files_count FROM db.my_table$manifests;

-- View table history
SELECT made_current_at, snapshot_id, is_current_ancestor FROM db.my_table$history;

-- View branch and tag references
SELECT name, type, snapshot_id FROM db.my_table$refs;
```

## Partitioning Strategy

### Guidelines

* **Partition tables with more than ~1 million records.** Unpartitioned large tables force full scans.
* **Use moderate cardinality.** Too high (e.g., per-sensor_id with 50,000 sensors) creates millions of tiny partitions that can't be compacted. Too low (e.g., single region) provides minimal pruning benefit.
* **Time-series data:** Partition by `day(event_time)` or `hour(event_time)`, not by raw timestamp.
* **Flink SQL limitation:** Flink DDL does not support hidden partitioning transforms like `day()`, `bucket()`, or `truncate()`. Use the DataStream/Catalog API for hidden partitions, or partition by a pre-computed column.

### Partition Evolution

Iceberg supports changing partition strategy without rewriting data:

```sql
ALTER TABLE my_table ADD PARTITION FIELD days(event_time);
```

New data uses the new scheme; old data remains readable under the old scheme. Queries automatically use the correct partition logic.

### Partition Columns in Equality Fields

When using upsert mode with HASH distribution on a partitioned table, the partition columns MUST be included in the equality fields. For example, if the table is partitioned by `(event_date, region)`, the equality fields must include both:

```java
.equalityFieldColumns(Arrays.asList("event_id", "event_date", "region"))
```

Failing to include partition columns causes incorrect upsert behavior — updates may not find the correct rows to delete.

## DDL Reference for Iceberg Tables in Flink SQL

### CREATE TABLE with Common Properties

```sql
CREATE TABLE `glue_catalog`.`my_db`.`my_table` (
    id BIGINT COMMENT 'unique id',
    event_time TIMESTAMP(6),
    data STRING NOT NULL,
    region STRING,
    event_date DATE,
    PRIMARY KEY (id, event_date, region) NOT ENFORCED
) PARTITIONED BY (event_date, region)
WITH (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy',
    'write.target-file-size-bytes' = '134217728',
    'write.upsert.enabled' = 'true',
    'write.delete.mode' = 'merge-on-read',
    'write.update.mode' = 'merge-on-read',
    'write.merge.mode' = 'merge-on-read'
);
```

### ALTER TABLE (Properties Only)

Flink only supports altering table properties, not columns or partitions:

```sql
ALTER TABLE my_table SET ('write.format.default' = 'orc');
ALTER TABLE my_table RENAME TO new_table_name;
```

### Flink DDL Limitations

* No hidden partitioning transforms (`day()`, `bucket()`, `truncate()`) in DDL — use DataStream API or pre-computed columns
* No computed columns
* No watermark definitions in DDL
* Column and partition changes not supported via ALTER TABLE

# Iceberg Tuning and Operations on Managed Service for Apache Flink

## Overview

This guide covers production tuning and operational concerns for Apache Iceberg tables written by Flink applications on Amazon Managed Service for Apache Flink (MSF): the small files problem, table maintenance (compaction, snapshot expiration, orphan file cleanup), catalog choice between AWS Glue and S3 Tables, monitoring, dependency management, and common anti-patterns.

For Iceberg write APIs, read patterns, partitioning, and DDL, see [iceberg-connector-guide.md](iceberg-connector-guide.md).

## The Small Files Problem

This is the #1 production issue with streaming Iceberg workloads. Understanding the math is critical.

### Three Root Causes of Small Files

**1. High commit rate (checkpoint interval):** Iceberg commits happen at Flink checkpoint boundaries. Files are closed at checkpoint boundaries regardless of whether they've reached the target size. The checkpoint interval is the single biggest lever for controlling file count.

```
Files per commit (worst case) = writer_parallelism × active_partitions
Files per day = (86400 / checkpoint_interval_seconds) × files_per_commit
```

Example with 60-second checkpoints, 4 writer tasks, 10 active partitions:

- 1,440 checkpoints/day × 4 × 10 = 57,600 files/day
- After 7 days: 403,200 files, each potentially only 1-10 MB

With a 10-second checkpoint interval, that becomes 345,600 files/day.

**2. MoR delete files from upserts:** Each upsert within a checkpoint generates both a data file (the new row) and an equality delete file (marking the old row for deletion). Upsert workloads create roughly 2× the files of append-only workloads. These delete files are typically tiny (just the equality field values) but accumulate and must be merged at read time.

**3. No shuffle before writing (distribution mode NONE):** Without distribution, each writer task creates one file per partition it touches per checkpoint. If data arrives in random order across all partitions, every writer touches every partition. With N writer tasks and M active partitions, you get up to N × M files per commit. HASH distribution routes each partition's data to a single writer, reducing to M files per commit — but is limited by partition cardinality (if you have 5 partitions, only 5 of 20 writer tasks get data). RANGE distribution addresses this by using traffic statistics to balance load across writers regardless of cardinality.

### Impact

- **Query planning:** Must read metadata for every file. 100,000 files can take 30-60 seconds just to plan a query.
- **Query execution:** Each file requires a separate S3 GET request. More files = more requests = higher latency and cost.
- **Metadata bloat:** Each commit generates manifest files. Thousands of commits create thousands of manifests tracking overlapping file sets.

### Mitigation Strategies

Apply these together — none is sufficient on its own for production streaming workloads.

1. **Increase checkpoint interval:** The single most effective lever. A 60-second interval creates 24x fewer files than a 2.5-second interval. On MSF, configure this at the application level (not in code):

   ```typescript
   // CDK
   checkpointConfiguration: {
       configurationType: 'CUSTOM',
       checkpointingEnabled: true,
       checkpointInterval: 60000,       // 60 seconds
       minPauseBetweenCheckpoints: 30000, // 30 seconds
   }
   ```

2. **Use table maintenance (compaction):** Merge small files into larger ones. See the Table Maintenance section below. This is required for production streaming Iceberg, not optional.

3. **Target file size:** Set `write.target-file-size-bytes` to 128-256 MB. The writer will try to reach this size before closing a file, but checkpoint boundaries force file closure regardless.

4. **Use RANGE or HASH distribution to reduce per-writer file creation:** RANGE distribution is recommended for skewed data; HASH distribution is required for upsert correctness on partitioned tables. Without a distribution mode, every writer can write to every partition, multiplying file count by writer parallelism. See the Distribution Modes section in [iceberg-connector-guide.md](iceberg-connector-guide.md) for details.

5. **Monitor actively:** Query the `$files` metadata table to track file sizes and counts. Alert when average file size drops below 32 MB or file count per partition exceeds 100.

### Snapshot Retention for Streaming

Default retention policies are designed for batch workloads. For streaming:

- Use **count-based retention** (e.g., retain last 100-1000 snapshots) rather than time-based (e.g., 7 days)
- A streaming job creating snapshots every 60 seconds generates 10,080 snapshots per week
- Keep active storage ratio above 85% (current data / total stored data)
- Compacted files leave behind orphaned old files — aggressive snapshot expiration is needed to clean them up
- **Long retention defeats the purpose of compaction.** Old data files cannot be physically removed (via `DeleteOrphanFiles`) while *any* retained snapshot still references them. A 7-day retention on a 60-second checkpoint job pins the post-compaction "old" files in S3 for the full 7 days, so storage footprint stays inflated even after compaction runs and you keep paying for the same bytes twice. Count-based retention solves this directly — once the retained snapshots roll past, compaction can actually reclaim space.

## Table Maintenance

### Overview

Iceberg tables require ongoing maintenance for production health. The three core operations, in correct execution order:

1. **Compact data files** (RewriteDataFiles) — Merge small files into larger ones
2. **Expire snapshots** (ExpireSnapshots) — Remove old table versions, orphaning old files
3. **Delete orphan files** (DeleteOrphanFiles) — Clean up files no longer referenced by any snapshot

**Running these out of order can cause data loss or corruption.** For example, expiring snapshots before compaction can orphan files that are still needed.

After all three, optionally:
4. **Rewrite manifests** — Consolidate metadata structure (not available in Flink streaming maintenance API, use Spark or batch Flink)

### Three Distinct Maintenance Approaches on AWS

There are exactly three ways to run maintenance for Iceberg tables written by Flink on AWS. The first decision is your catalog (S3 Tables vs Glue), and if you pick Glue, the second decision is which maintenance mechanism to use:

| Approach | Catalog | Compaction | Snapshot expiration | Orphan cleanup | Operational overhead | Control |
|---|---|---|---|---|---|---|
| **1. S3 Tables (fully managed)** | S3 Tables | Automatic | Automatic | Automatic | None | Low — service overrides some table properties |
| **2. Glue + Glue auto-compaction** | Glue | Managed by Glue | You handle (Flink or external) | You handle (Flink or external) | Medium — only snapshot/orphan cleanup to run | Medium — Glue manages compaction thresholds |
| **3. Glue + Flink embedded maintenance** | Glue | Flink job topology | Flink job topology | Flink job topology | High — RDS for JDBC locks, VPC config | Full — every parameter is yours to tune |

**Key constraints (do not violate):**

- S3 Tables: do NOT add Flink embedded maintenance or external compaction. Concurrent maintenance causes commit conflicts.
- Glue: do NOT combine Glue auto-compaction with Flink embedded compaction on the same table. Pick one compaction mechanism. (You can still pair Glue auto-compaction with Flink embedded snapshot expiration and orphan cleanup — those are not redundant.)

**Quick picker:**

- Want zero maintenance work and accept S3 Tables' constraints? → S3 Tables
- Want Glue catalog (broader query engine support, full table-property control) but don't want to operate compaction yourself? → Glue + Glue auto-compaction
- Need to control compaction strategy, scheduling, and partial-progress behavior precisely? → Glue + Flink embedded maintenance

These approaches are detailed in the next section, followed by the Flink TableMaintenance API used by Glue + Flink embedded maintenance (and for the snapshot/orphan portions of Glue + Glue auto-compaction).

### The Three Alternatives in Detail

Each of the three approaches introduced above is described below with its specific behaviors and pitfalls.

**S3 Tables (fully managed):**

- Compaction is automatic and enabled by default. Target file size: 512 MB (configurable 64-512 MB). Strategies: auto (default), binpack, sort, z-order.
- Compaction applies delete file effects — merges equality/position deletes into data files automatically.
- Snapshot management is automatic: defaults to min 1 snapshot, max 120 hours age. Configurable via `PutTableMaintenanceConfiguration` API.
- Unreferenced file removal is automatic.
- Do NOT run Flink embedded maintenance or external compaction alongside S3 Tables — it will cause commit conflicts with the service's own maintenance.
- Limitation: S3 Tables overrides some table properties. S3 Tables snapshot management does NOT respect Iceberg table properties set via `ALTER TABLE SET TBLPROPERTIES` (e.g., branch/tag retention). If you set such properties, S3 Tables disables its own snapshot management and you must handle it yourself.
- Transient commit conflicts between S3 Tables compaction and your streaming writer are normal — S3 Tables handles retry internally, but you may see transient errors in Flink logs.

**Glue Catalog with Glue Auto-Compaction (managed compaction, manual snapshot/orphan cleanup):**

- AWS Glue Data Catalog supports automatic compaction for Iceberg tables. It monitors partitions and triggers compaction when thresholds are met (e.g., >100 files smaller than 75% of target size).
- Supports both CoW and MoR tables, including compacting delete files.
- Commits partial progress regularly.
- You still need to handle snapshot expiration and orphan file cleanup yourself — use Flink's TableMaintenance API for those, or schedule external jobs.
- Concurrent write conflicts between Glue compaction and your streaming writer are possible. Glue handles retries, but your Flink job should tolerate transient commit failures.

**Glue Catalog with Flink Embedded Maintenance (full control):**

- Full control over all three operations: compaction, snapshot expiration, orphan cleanup.
- Runs inside the Flink job topology, coordinated by distributed locks (JDBC/ZK). No external compaction conflicts.
- Requires infrastructure: RDS PostgreSQL instance for JDBC locks, VPC configuration for the Flink app.
- Most flexible but most operational overhead.
- Do NOT combine with Glue auto-compaction — pick one compaction approach to avoid conflicts.

**Decision guide (recap):**

- Want zero maintenance overhead? → S3 Tables (Approach 1)
- Want managed compaction but keep Glue catalog flexibility? → Glue auto-compaction + Flink for snapshot/orphan cleanup (Approach 2)
- Need full control over maintenance scheduling and parameters? → Flink embedded maintenance with JDBC locks on Glue Catalog (Approach 3)

### Flink Streaming Maintenance (TableMaintenance API)

The TableMaintenance API (Iceberg 1.10+) runs maintenance as part of the Flink job topology, triggered by post-commit events. Requires IcebergSink (SinkV2).

Store the JDBC lock-database credentials in AWS Secrets Manager and look them up at job startup. **Never hardcode credentials in application code, runtime properties, or `setup.sql`/JAR resources.** Connect to the lock database over TLS — the JDBC URL must include `ssl=true` so the connection is encrypted in transit. Certificate verification (`sslmode=verify-full`) on MSF requires a custom `SSLSocketFactory` that loads the CA bundle from the classpath, since MSF doesn't expose a stable filesystem path for `sslrootcert`; see [cdc-connector-guide.md](cdc-connector-guide.md#tls--ssl-to-the-database) for the constraints. See [cdc-connector-guide.md](cdc-connector-guide.md#database-credentials-and-secrets-management) for the full Secrets Manager pattern; the Iceberg lock-DB credentials should follow the same approach.

```java
// Resolve credentials from AWS Secrets Manager — see cdc-connector-guide.md for the
// full SecretsManagerClient pattern and IAM grant.
DbCreds lockDbCreds = loadDbCreds(cdcConfig.getProperty("iceberg.lock.secret.id"));

Map<String, String> jdbcProps = new HashMap<>();
jdbcProps.put("jdbc.user", lockDbCreds.username);
jdbcProps.put("jdbc.password", lockDbCreds.password);
jdbcProps.put("flink-maintenance.lock.jdbc.init-lock-tables", "true");

TriggerLockFactory lockFactory = new JdbcLockFactory(
    "jdbc:postgresql://rds-endpoint:5432/iceberg_locks?ssl=true",
    "catalog.database.table",  // Unique lock ID per table
    jdbcProps
);

TableMaintenance.forTable(env, tableLoader, lockFactory)
    .uidSuffix("my-maintenance")
    .rateLimit(Duration.ofMinutes(10))        // Min interval between executions
    .lockCheckDelay(Duration.ofSeconds(30))   // Delay before checking lock availability
    .add(ExpireSnapshots.builder()
        .scheduleOnCommitCount(10)            // Trigger after every 10 commits
        .maxSnapshotAge(Duration.ofHours(24))
        .retainLast(5)
        .deleteBatchSize(1000))
    .add(RewriteDataFiles.builder()
        .scheduleOnDataFileCount(20)          // Trigger when 20+ small files exist
        .targetFileSizeBytes(256 * 1024 * 1024)  // 256 MB target
        .minFileSizeBytes(32 * 1024 * 1024)      // Files below 32 MB are candidates
        .partialProgressEnabled(true)             // Commit progress incrementally
        .partialProgressMaxCommits(5)
        .maxRewriteBytes(2L * 1024 * 1024 * 1024))  // Cap at 2 GB per run
    .add(DeleteOrphanFiles.builder()
        .scheduleOnCommitCount(50)            // Less frequent than compaction
        .minAge(Duration.ofDays(3)))          // Only delete files older than 3 days
    .append();
```

### Post-Commit Maintenance via IcebergSink Configuration

Alternative to the explicit TableMaintenance API — configure maintenance directly on the sink:

```java
Map<String, String> flinkConf = new HashMap<>();
flinkConf.put(FlinkWriteOptions.COMPACTION_ENABLE.key(), "true");
flinkConf.put(LockConfig.LOCK_TYPE_OPTION.key(), LockConfig.JdbcLockConfig.JDBC);
flinkConf.put(LockConfig.JdbcLockConfig.JDBC_URI_OPTION.key(),
    "jdbc:postgresql://host:5432/iceberg?ssl=true");
flinkConf.put(LockConfig.LOCK_ID_OPTION.key(), "catalog.db.table");
// Lock-DB user/password must be supplied via Secrets Manager — do not hardcode.
// e.g.: flinkConf.put(LockConfig.JdbcLockConfig.JDBC_USER_OPTION.key(), lockDbCreds.username);

IcebergSink.forRowData(dataStream)
    .tableLoader(tableLoader)
    .setAll(flinkConf)
    .append();
```

Or via SQL:

```sql
SET 'table.exec.iceberg.use-v2-sink' = 'true';
SET 'compaction-enabled' = 'true';
SET 'flink-maintenance.lock.type' = 'jdbc';
SET 'flink-maintenance.lock.lock-id' = 'catalog.db.table';
SET 'flink-maintenance.lock.jdbc.uri' = 'jdbc:postgresql://host:5432/iceberg?ssl=true';
SET 'flink-maintenance.lock.jdbc.init-lock-tables' = 'true';
-- jdbc.user / jdbc.password must come from a Secrets Manager lookup performed
-- in main() and templated into the SET statement before submission. Do NOT
-- store them in MSF runtime properties (even via {{resolve:secretsmanager:...}}
-- dynamic references) — they would land as plaintext on the deployed property
-- surface. See cdc-connector-guide.md → Database Credentials and Secrets Management.

INSERT INTO my_table SELECT ...;
```

### Lock Factory Options

Maintenance requires distributed locks to prevent concurrent operations on the same table:

| Lock Type | When to Use | Infrastructure Required |
|---|---|---|
| JDBC (PostgreSQL) | Most MSF deployments | RDS PostgreSQL instance, VPC for Flink app |
| ZooKeeper | If ZK is already available | ZooKeeper cluster |

JDBC lock factory with auto-table creation:

```java
// Resolve lock-DB credentials from Secrets Manager — do not hardcode (see
// cdc-connector-guide.md → Database Credentials and Secrets Management).
DbCreds lockDbCreds = loadDbCreds(cdcConfig.getProperty("iceberg.lock.secret.id"));

Map<String, String> jdbcProps = new HashMap<>();
jdbcProps.put("jdbc.user", lockDbCreds.username);
jdbcProps.put("jdbc.password", lockDbCreds.password);
jdbcProps.put("flink-maintenance.lock.jdbc.init-lock-tables", "true");

// jdbcUrl should enforce TLS: jdbc:postgresql://host:5432/db?ssl=true
// (sslmode=verify-full requires a custom SSLSocketFactory on MSF — see
// cdc-connector-guide.md → TLS / SSL to the database)
TriggerLockFactory lockFactory = new JdbcLockFactory(jdbcUrl, lockId, jdbcProps);
lockFactory.open();  // Initialize lock tables
```

### Scheduling Triggers

Choose triggers based on your workload:

| Trigger | Method | Best For |
|---|---|---|
| Commit count | `scheduleOnCommitCount(N)` | Write-heavy tables with frequent commits |
| Data file count | `scheduleOnDataFileCount(N)` | Fine-grained control over small file accumulation |
| Data file size | `scheduleOnDataFileSize(bytes)` | Size-based thresholds |
| Delete file count | `scheduleOnEqDeleteFileCount(N)` | Upsert tables with equality delete accumulation |
| Time interval | `scheduleOnInterval(Duration)` | Regular cadence regardless of write activity |

### Maintenance Troubleshooting

- **OutOfMemoryError during file deletion:** Reduce `deleteBatchSize` (e.g., from 1000 to 500)
- **Lock conflicts between jobs:** Increase `lockCheckDelay` and `rateLimit`
- **Compaction can't keep up:** Enable `partialProgressEnabled`, set `maxRewriteBytes` to cap work per run, increase compaction parallelism
- **Orphan file cleanup safety:** The Flink streaming writer stores uncommitted data as temporary files. Set `minAge` to at least 3 days to avoid deleting files from in-progress checkpoints. Also keep the last snapshot created by the Flink job (identifiable by `flink.job-id` in snapshot summary).

## Catalog Configuration on AWS

### AWS Glue Catalog (DataStream API)

```java
Map<String, String> catalogProps = new HashMap<>();
catalogProps.put("catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog");
catalogProps.put("io-impl", "org.apache.iceberg.aws.s3.S3FileIO");
catalogProps.put("warehouse", "s3://my-bucket/warehouse");
catalogProps.put("client.region", "us-east-1");
catalogProps.put("glue.region", "us-east-1");

CatalogLoader catalogLoader = CatalogLoader.custom(
    "glue_catalog", catalogProps, new Configuration(),
    "org.apache.iceberg.aws.glue.GlueCatalog"
);
```

### AWS Glue Catalog (SQL)

```sql
CREATE CATALOG glue_catalog WITH (
    'type' = 'iceberg',
    'catalog-impl' = 'org.apache.iceberg.aws.glue.GlueCatalog',
    'io-impl' = 'org.apache.iceberg.aws.s3.S3FileIO',
    'warehouse' = 's3://my-bucket/warehouse',
    'glue.skip-archive' = 'true',
    'glue.skip-name-validation' = 'true'
);
```

### S3 Tables Catalog (DataStream API)

```java
Map<String, String> catalogProps = new HashMap<>();
catalogProps.put("catalog-impl", "software.amazon.s3tables.iceberg.S3TablesCatalog");
catalogProps.put("warehouse", s3TableBucketArn);  // ARN, not S3 path
catalogProps.put("client.region", "us-east-1");
catalogProps.put("s3tables.catalog.client.region", "us-east-1");

CatalogLoader catalogLoader = CatalogLoader.custom(
    "s3tables_catalog", catalogProps, new Configuration(),
    "software.amazon.s3tables.iceberg.S3TablesCatalog"
);
```

### S3 Tables Catalog (SQL)

```sql
CREATE CATALOG s3tables_catalog WITH (
    'type' = 'iceberg',
    'catalog-impl' = 'software.amazon.s3tables.iceberg.S3TablesCatalog',
    'warehouse' = 'arn:aws:s3tables:us-east-1:123456789012:bucket/my-table-bucket',
    'client.region' = 'us-east-1'
);
```

### Glue vs S3 Tables Decision Guide

| Aspect | Glue Catalog | S3 Tables |
|---|---|---|
| Compaction | Manual (Flink embedded or Glue auto-compaction) | Automatic (binpack, sort, z-order strategies; target 64-512 MB) |
| Snapshot expiration | Manual (Flink embedded or external jobs) | Automatic (default: min 1 snapshot, max 120h age; configurable) |
| Orphan file cleanup | Manual | Automatic |
| Delete file compaction | Glue auto-compaction handles MoR delete files; or Flink embedded | Automatic (applies delete effects during compaction) |
| Query engine support | Broad (Athena, Spark, Trino, Redshift, EMR) | Growing support |
| Storage | Standard S3 bucket (you manage lifecycle, encryption) | S3 Table Bucket (managed) |
| Cost | S3 storage + Glue API calls + compute for maintenance | S3 Tables pricing (includes maintenance compute) |
| Control | Full control over table properties, maintenance scheduling, retention | Less control; S3 Tables overrides some Iceberg table properties |
| Branch/tag retention | Fully supported via Iceberg table properties | Setting branch/tag retention disables S3 Tables snapshot management |

**Rule:** Do NOT enable Flink embedded maintenance or external compaction when using S3 Tables — it handles this automatically and concurrent maintenance causes commit conflicts.

### Flink Connector-Style Catalog (SQL)

For simple use cases, create Iceberg tables directly without a named catalog:

```sql
CREATE TABLE my_table (
    id BIGINT,
    data STRING
) WITH (
    'connector' = 'iceberg',
    'catalog-name' = 'glue_prod',
    'catalog-impl' = 'org.apache.iceberg.aws.glue.GlueCatalog',
    'warehouse' = 's3://my-bucket/warehouse'
);
```

This creates a Flink table in the default Flink catalog that maps to the underlying Iceberg table. The Iceberg catalog is configured inline via table properties.

## Monitoring Iceberg Workloads

### Key Sink Metrics

IcebergSink exposes Flink metrics under `IcebergStreamWriter` and `IcebergFilesCommitter` sub-groups:

| Metric | Type | What to Monitor |
|---|---|---|
| `elapsedSecondsSinceLastSuccessfulCommit` | Gauge | **Primary alerting metric.** If checkpoint interval is 60s, alert when this exceeds 600s (10 minutes). Detects failed or missing Iceberg commits. |
| `lastFlushDurationMs` | Gauge | Time to flush and upload files during checkpoint. Increasing values indicate growing file counts or S3 latency. |
| `lastCommitDurationMs` | Gauge | Time for the Iceberg table commit. Increasing values indicate metadata bloat. |
| `committedDataFilesCount` | Counter | Track rate of file creation. High rates indicate small file accumulation. |
| `committedDeleteFilesCount` | Counter | Track delete file accumulation in upsert workloads. |
| `dataFilesSizeHistogram` | Histogram | Distribution of data file sizes. Median should be near target file size. |

### Metadata Table Monitoring Queries

Run these periodically (e.g., via Athena) to assess table health:

```sql
-- Small files detection: count files below 32 MB
SELECT COUNT(*) as small_file_count,
       AVG(file_size_in_bytes) as avg_size,
       MIN(file_size_in_bytes) as min_size
FROM db.my_table$files
WHERE file_size_in_bytes < 33554432;

-- Delete file accumulation per partition
SELECT partition,
       equality_delete_file_count,
       equality_delete_record_count,
       file_count
FROM db.my_table$partitions
WHERE equality_delete_file_count > 10;

-- Snapshot velocity (how fast are we creating snapshots)
SELECT COUNT(*) as snapshot_count
FROM db.my_table$snapshots
WHERE committed_at > CURRENT_TIMESTAMP - INTERVAL '1' HOUR;
```

## Dependency Management for Iceberg on MSF

### Maven Dependencies (Flink 1.20)

```xml
<properties>
    <flink.version>1.20.3</flink.version>
    <iceberg.version>1.10.0</iceberg.version>
    <aws.sdk.version>2.33.0</aws.sdk.version>
    <hadoop.version>3.4.0</hadoop.version>
</properties>

<dependencies>
    <!-- Iceberg Flink Runtime (shaded bundle) -->
    <dependency>
        <groupId>org.apache.iceberg</groupId>
        <artifactId>iceberg-flink-runtime-1.20</artifactId>
        <version>${iceberg.version}</version>
    </dependency>

    <!-- Iceberg AWS Bundle (Glue Catalog, S3FileIO) -->
    <dependency>
        <groupId>org.apache.iceberg</groupId>
        <artifactId>iceberg-aws-bundle</artifactId>
        <version>${iceberg.version}</version>
    </dependency>

    <!-- S3 Tables Catalog (only if using S3 Tables) -->
    <dependency>
        <groupId>software.amazon.s3tables</groupId>
        <artifactId>s3-tables-catalog-for-iceberg</artifactId>
        <version>0.1.8</version>
    </dependency>

    <!-- Hadoop Common — required by Iceberg's CatalogLoader API
         (org.apache.hadoop.conf.Configuration is referenced by
         CatalogLoader.custom(name, props, new Configuration(), implClass)
         at compile and run time). Neither iceberg-flink-runtime-* nor
         iceberg-aws-bundle brings it transitively (they are shaded uber-jars),
         and MSF does not ship hadoop-common on the application classpath.
         Compile scope so it ends up in the shaded JAR under the
         shaded.org.apache.hadoop.conf relocation.

         Exclusions: hadoop-common transitively pulls slf4j-reload4j, reload4j,
         and log4j 1.x. These conflict with log4j-slf4j-impl (Log4j 2.x) that
         the MSF runtime expects; without these exclusions you get duplicate
         SLF4J bindings and classloading failures at startup. -->
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>${hadoop.version}</version>
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-reload4j</artifactId>
            </exclusion>
            <exclusion>
                <groupId>ch.qos.reload4j</groupId>
                <artifactId>reload4j</artifactId>
            </exclusion>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <!-- Kinesis Connector -->
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-connector-aws-kinesis-streams</artifactId>
        <version>5.0.0-1.20</version>
    </dependency>

    <!-- Flink Table API (required for SQL path) -->
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-table-api-java-bridge</artifactId>
        <version>${flink.version}</version>
        <scope>provided</scope>
    </dependency>

    <!-- MSF Runtime -->
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-kinesisanalytics-runtime</artifactId>
        <version>1.2.0</version>
        <scope>provided</scope>
    </dependency>
</dependencies>
```

### Critical Shade Plugin Configuration

MSF bundles its own Hadoop and AWS SDK classes. You MUST relocate conflicting classes to avoid classpath conflicts:

```xml
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <configuration>
        <relocations>
            <!-- Relocate Hadoop conf to avoid conflict with flink-s3-fs-hadoop -->
            <relocation>
                <pattern>org.apache.hadoop.conf</pattern>
                <shadedPattern>shaded.org.apache.hadoop.conf</shadedPattern>
            </relocation>
            <!-- Relocate AWS SDK v2 to avoid conflict with MSF's bundled SDK -->
            <relocation>
                <pattern>software.amazon.awssdk</pattern>
                <shadedPattern>shaded.software.amazon.awssdk</shadedPattern>
            </relocation>
        </relocations>
        <transformers>
            <!-- CRITICAL: Required for SPI service discovery (Iceberg FileIO, Catalog) -->
            <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
            <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                <mainClass>com.example.MyFlinkJob</mainClass>
            </transformer>
        </transformers>
    </configuration>
</plugin>
```

**The `ServicesResourceTransformer` is essential.** Without it, Iceberg's SPI-based service discovery (for FileIO implementations, catalog implementations) will fail at runtime with ClassNotFoundException.

**The `org.apache.hadoop.conf` relocation requires `hadoop-common` as a compile-scope dependency.** The relocation rewrites references in your shaded JAR from `org.apache.hadoop.conf.*` to `shaded.org.apache.hadoop.conf.*` so they don't collide with classes loaded by `flink-s3-fs-hadoop` on MSF — but the relocation only has anything to rewrite if `hadoop-common` is actually in the shade input. If you call `CatalogLoader.custom(name, props, new Configuration(), implClass)` (or otherwise reference `org.apache.hadoop.conf.Configuration`) without adding `hadoop-common` to the pom, compilation fails with `package org.apache.hadoop.conf does not exist`. Adding `hadoop-common` and applying the relocation are a pair: the dep brings the class in, the relocation keeps it from colliding with the MSF-bundled copy.

## Common Anti-Patterns

1. **No compaction strategy for streaming writes.** Small files accumulate silently until queries become unusable. Always pair streaming writes with maintenance (S3 Tables auto-maintenance, Glue auto-compaction, or Flink embedded maintenance).

2. **Time-based snapshot retention for streaming.** "Keep 7 days" means 600,000+ snapshots at 60-second intervals. Use count-based retention.

3. **Enabling maintenance with S3 Tables.** S3 Tables handles compaction, snapshot management, and orphan cleanup automatically. Running Flink embedded maintenance or external compaction alongside it causes commit conflicts.

4. **Missing partition columns in equality fields.** Causes incorrect upsert behavior with HASH distribution on partitioned tables.

5. **Configuring checkpoints in application code for MSF.** MSF manages checkpointing. Only configure checkpoints in code for local development.

6. **Using FlinkSink when maintenance is needed.** The TableMaintenance API requires IcebergSink (SinkV2). FlinkSink does not support post-commit maintenance topology.

7. **Streaming reads from upsert tables.** IcebergSource streaming mode only supports append-only tables. Tables with equality deletes are not supported for streaming reads.

8. **Running maintenance operations out of order.** The correct order is: compact → expire snapshots → delete orphans → rewrite manifests. Running orphan cleanup before snapshot expiration can delete files still referenced by active snapshots.

9. **High-cardinality partitioning.** Partitioning by a column with millions of distinct values creates millions of tiny partitions that can't be compacted within partition boundaries.

10. **Missing `ServicesResourceTransformer` in shade plugin.** Causes runtime ClassNotFoundException for Iceberg FileIO and Catalog implementations on MSF.

11. **Using distribution mode NONE with upsert.** Without HASH distribution, Flink uses rebalance to distribute records across writer tasks. Multiple updates to the same key within a checkpoint can land on different writers, causing the delete file on one writer to miss the insert on another — resulting in duplicate rows. Always use `distributionMode(DistributionMode.HASH)` for upsert workloads.

12. **Assuming multi-table writes are atomic.** Iceberg commits are per-table. When writing to multiple tables via StatementSet, each table commits independently. Failure between commits leaves tables in inconsistent state. Design downstream consumers to tolerate this, or use snapshot correlation via `flink.job-id` in snapshot summary.

13. **Combining Glue auto-compaction with Flink embedded compaction.** Both will attempt to compact the same files, causing commit conflicts and wasted compute. Pick one compaction approach per table.

# Job Graph Anti-Patterns Guide

## Overview

This guide covers common job graph anti-patterns for Managed Service for Apache Flink applications: data skew detection and mitigation, the monolith job anti-pattern, and the high fan-out anti-pattern. Use it when diagnosing performance problems or deciding whether to split a large application.

For operator chaining, operator-to-task-slot mapping, and task slot overload guidance, see [job-graph-architecture.md](job-graph-architecture.md).

## Data Skew Detection and Mitigation

### Identifying Skew via Flink Web UI

Data skew occurs when some subtasks of a keyed operator receive significantly more data than others. This causes the overloaded subtasks to become bottlenecks while other subtasks sit idle.

**How to detect skew in the Flink Web UI:**

1. Open the running job → click on a keyed operator (any operator after a `keyBy`)
2. Select the "Subtasks" tab
3. Compare these columns across subtasks:
   - `Bytes Received`: should be roughly equal across subtasks
   - `Records Received`: should be roughly equal across subtasks
   - `Busy Time (ms/s)`: skewed subtasks show much higher values
4. If the max value is more than 3× the median, the operator has significant skew

A subtask processing 10× more records than its peers will become the throughput bottleneck for the entire operator, regardless of how many other subtasks are idle.

### Programmatic Detection

For automated checks (alarms, scheduled diagnostics), pull the same per-subtask data without the UI:

- **Flink Dashboard REST API** — `/jobs/$JOB_ID/vertices/$VERTEX_ID/subtasks` returns the same `read-records` / `read-bytes` / `busyTimeMsPerSecond` per-subtask values shown in the UI. See the table in [first-fault-isolation.md](first-fault-isolation.md) for related per-vertex endpoints (backpressure, watermarks). Useful for scripted skew detection where the rule of thumb is `max / median > 3`.
- **CloudWatch metrics at the `PARALLELISM` dimension level** — set `MonitoringConfiguration.MetricsLevel` to `PARALLELISM` (or `OPERATOR`) so per-subtask metrics flow to CloudWatch with a `subtaskIndex` dimension. You can then alarm on per-subtask `numRecordsInPerSecond` variance. Note that higher metric levels increase CloudWatch metric cardinality and cost — see the metric-level guidance in [monitoring-and-metrics.md](monitoring-and-metrics.md) before enabling on a high-parallelism job.

Confirm and quantify skew before changing parallelism. Lowering parallelism is a valid mitigation only after the per-subtask numbers above show that most subtasks are idle while a few are saturated; doing it preemptively can mask the underlying hot-key problem.

### Diagnostic Checklist for Uneven Processing

1. **Key distribution**: Check for "hot" keys concentrating traffic on a few subtasks. Query source data for key cardinality and frequency distribution.
2. **Partition assignment**: Check if Kinesis shards/Kafka partitions are evenly sized. Uneven source partitions propagate imbalance downstream.
3. **Hash collisions**: Poor `hashCode()` implementations can map many distinct keys to the same subtask. Verify custom key types distribute evenly.
4. **Key cardinality vs parallelism**: Ensure key cardinality is at least 10× the operator parallelism for reasonable distribution.
5. **Temporal skew**: Some keys may be hot only during certain time windows. Check if skew is sustained or periodic.

### Warning: High Core Counts + Kryo Serialization + Data Skew

Running Managed Service for Apache Flink applications with high KPU counts (e.g., 64+ KPUs) combined with Kryo serialization and data skew creates a compounding performance problem:

- **Kryo serialization** is slower and produces larger serialized objects than POJO or Avro serialization. This increases network transfer time during shuffles (`keyBy`, `rebalance`).
- **Data skew** concentrates traffic on a few subtasks, which must deserialize a disproportionate share of Kryo-encoded records.
- **High parallelism** amplifies the shuffle: with 64 subtasks, each record after a `keyBy` is serialized, sent over the network, and deserialized. The skewed subtasks become CPU-bound on Kryo deserialization.

**Remediation:**

- Switch from Kryo to POJO serialization by ensuring your data classes follow Flink's POJO rules (public class, public no-arg constructor, public fields or getters/setters). Check logs for `"Class ... cannot be used as a POJO type"` to find Kryo fallbacks.
- If POJO is not feasible, use Avro serialization with a defined schema.
- Address the skew itself: add a salt/prefix to hot keys to spread them across subtasks, then aggregate in a second pass.
- Only after skew is confirmed and quantified per the steps above, consider reducing parallelism for the skewed operator if most subtasks are idle while a few are saturated. Do not lower parallelism preemptively — it masks the underlying hot-key problem and can hurt unrelated operators in the same job.

## Monolith Job Anti-Pattern

### Indicators of a Monolith Job

A monolith Flink job tries to do too much in a single application. Watch for these indicators:

- **Excessive operator count**: More than 100–150 distinct operators in the job graph. Each operator adds state management, checkpoint overhead, and metric cardinality.
- **Unrelated business logic paths**: The job reads from multiple sources and processes them through independent pipelines that never join or share state. These are separate applications forced into one deployment.
- **Parallelism exceeding 100 per KPU**: When the application needs very high parallelism to keep up, it often means multiple workloads with different throughput requirements are bundled together.
- **Mixed SLA requirements**: Part of the job needs sub-second latency while another part is a batch-like hourly aggregation. These cannot be optimally tuned in a single application.
- **Frequent full-job restarts for partial changes**: A code change to one processing path requires restarting the entire application, including unrelated paths that lose their state or must restore from checkpoint.

### Criteria for Splitting

Split a monolith job when any of these conditions are met:

1. **Independent processing paths**: Two or more pipelines in the job share no state and no data exchange. They are logically separate applications.
2. **Different scaling requirements**: One path needs 32 KPUs to handle peak traffic while another needs only 4. Bundling them wastes resources on the smaller path or under-provisions the larger one.
3. **Different SLA requirements**: One path has a strict latency SLA (< 1 second end-to-end) while another is latency-tolerant (minutes). Checkpoint intervals, parallelism, and resource allocation cannot be optimized for both simultaneously.
4. **Blast radius reduction**: A failure or bug in one processing path should not take down unrelated paths. Separate applications isolate failures.
5. **Independent deployment cycles**: Different teams own different processing paths and need to deploy changes independently.

### Splitting Patterns

**Pattern: Shared Kinesis stream as intermediate topic** — Split the monolith into a producer job and consumer jobs connected by a Kinesis Data Stream. Each job scales independently; if the consumer fails, the producer continues and the consumer replays on recovery.

**Pattern: Idempotent sinks for exactly-once across jobs** — Each job maintains its own checkpoint. Use idempotent sinks (upsert to DynamoDB, deduplication keys) to handle duplicates at the boundary. The intermediate stream provides at-least-once delivery.

### Trade-Offs of Job Splitting

| Factor | Monolith | Split Jobs |
|--------|----------|------------|
| Blast radius | One failure affects everything | Failures isolated |
| Latency | Lower (no intermediate hop) | Higher (extra serialization) |
| Scaling | One-size-fits-all KPU allocation | Each job sized independently |
| Deployment | Full restart for any change | Only affected job restarts |
| Cost | Single application | Multiple applications + intermediate stream costs |

**General guidance**: Split when the monolith causes operational pain (frequent restarts, resource waste, mixed SLAs). For small applications with < 50 operators and uniform requirements, a single job is simpler.

## High Fan-Out Anti-Pattern

### The One-Sink-Per-Shard/Partition Pattern

A common mistake is creating a separate sink operator for each output shard or partition. For example, a developer routing events to 64 Kinesis shards might create 64 individual `KinesisStreamsSink` instances, each writing to a specific shard:

```java
// AVOID: Creating one sink per shard
for (int i = 0; i < 64; i++) {
    final int shardIndex = i;
    events.filter(e -> e.getShardKey() % 64 == shardIndex)
          .sinkTo(createKinesisSink("output-stream"))
          .uid("sink-shard-" + shardIndex);
}
```

This creates 64 sink operators in the job graph, each with its own parallelism, state, checkpoint overhead, and connection pool. The job graph balloons in complexity.

### Resource and Performance Consequences

Excessive sink operators cause cascading problems: thread exhaustion (each sink has its own thread pool), connection pool saturation (hundreds of concurrent connections per TaskManager), checkpoint size inflation (each sink maintains its own state), metric cardinality explosion (64 sinks = 64× the metrics), and operator-to-task-slot overload (pushing toward the >200 operator threshold discussed in [job-graph-architecture.md](job-graph-architecture.md)).

### Alternative Patterns

**Use a single partitioned sink with a partition key generator:**

```java
// RECOMMENDED: Single sink with built-in partitioning
KinesisStreamsSink<Event> sink = KinesisStreamsSink.<Event>builder()
    .setStreamName("output-stream")
    .setSerializationSchema(new EventSerializationSchema())
    .setPartitionKeyGenerator(event -> event.getPartitionKey())
    .build();

events.sinkTo(sink)
    .setParallelism(16)  // Match KPU count, not shard count
    .uid("kinesis-sink-uid");
```

The Kinesis sink handles partitioning internally using the partition key generator. One sink operator distributes records across all shards without creating separate operators per shard. **Set the parallelism to match your KPU count, not your shard/partition count** — the sink will still write to every shard regardless of operator parallelism, because Kinesis routes by partition-key hash, not by sender.

The same principle applies to Kafka sinks — use a single `KafkaSink` with a `KafkaRecordSerializationSchema` that sets the partition (or relies on the configured `Partitioner`), rather than creating one sink per partition. Same applies to JDBC, DynamoDB, and most other Flink sinks: one sink + a routing function inside the sink, not one sink per output bucket.

**Stay within the rule-of-thumb upper bound of 2–4 sink operators per KPU** — see the table in the next section for sizing. Crossing this number is a strong sign you're hitting the high fan-out anti-pattern even if no single sink is per-shard.

### Maximum Recommended Sink Count

**Rule of thumb: no more than 2–4 sink operators per KPU.**

| KPU Count | Max Recommended Sinks | Reasoning |
|-----------|----------------------|-----------|
| 4 | 8–16 | Each sink adds connection overhead and checkpoint state |
| 16 | 32–64 | Beyond this, connection pool and thread pressure becomes significant |
| 32 | 64–128 | At this scale, prefer fewer sinks with built-in partitioning |

If the application genuinely needs to write to many distinct destinations (different streams, different tables), consider:

- Using a single sink with a routing function that directs records to different targets based on record content
- Splitting into multiple jobs, each responsible for a subset of destinations (see Monolith Job Anti-Pattern above)
- Using side outputs to route records to a small number of categorized sinks rather than one per destination

# Job Graph Architecture Guide

## Overview

This guide covers Flink job graph design for Managed Service for Apache Flink applications: operator chaining, operator-to-task-slot mapping, and task slot overload diagnosis. Use it when designing a new job graph, diagnosing performance problems in an existing one, or deciding whether to split a large application. For anti-patterns (data skew, monolith jobs, high fan-out), see [job-graph-anti-patterns.md](job-graph-anti-patterns.md).

## Operator Chaining and Job Graph Basics

### How Operator Chaining Works

Flink automatically chains operators that meet all of these conditions into a single task:

- Same parallelism
- Connected by a forward (non-shuffle) data exchange
- Same slot sharing group
- Neither operator has chaining explicitly disabled

Chained operators run in the same thread, eliminating serialization/deserialization overhead and thread context switching between them. In the Flink Web UI, chained operators appear as a single box in the job graph with names joined by arrows (e.g., `Source → Map → Filter`).

**Verifying chaining in the Flink Web UI:**

1. Open the Flink Web UI from the Managed Service for Apache Flink console
2. Navigate to the running job's "Overview" tab
3. Each box in the job graph represents one task (a chain of operators)
4. Click a task box to see which operators are chained inside it
5. If operators you expected to be chained appear as separate boxes, check that parallelism matches and the data exchange is forward

**Diagnosing unexpected chain breaks** — work this list in order before assuming a bug or re-running the job:

1. **Parallelism mismatch.** Confirm every operator in the supposed chain runs at the same parallelism. Source connectors are a frequent culprit — a Kinesis source defaults to one subtask per shard, so a 4-shard stream can't chain with downstream operators set to 8. Per-operator `setParallelism()` overrides the env default.
2. **Implicit shuffle.** Any `keyBy()`, `rebalance()`, `shuffle()`, `broadcast()`, or `rescale()` between two operators inserts a network exchange and breaks the chain at that point. The exchange-label arrow (`HASH`, `REBALANCE`, `FORWARD`) between job-graph boxes tells you which case applies.
3. **`startNewChain()` or `disableChaining()` left in code.** These are commonly added for diagnostic isolation (see below) and forgotten.
4. **Different slot sharing groups.** A `slotSharingGroup("...")` on one operator that differs from its neighbor will prevent chaining even if everything else aligns.

Breaking a chain is not always wrong — see "Using `disableChaining()` and `startNewChain()` Strategically" below for legitimate reasons (operator-metric isolation, external-call latency separation, explicit parallelism boundaries). The Flink Web UI is the source of truth here, not "the display might be misleading."

### Operator-to-Task-Slot Mapping

Each task slot runs one parallel pipeline of chained operators. The number of operators per task slot depends on how many operators chain together and how many slot sharing groups exist.

**Rule of thumb: 20–40 operators per task slot.** This range balances resource utilization against overhead:

| Operators per Task Slot | Behavior |
|------------------------|----------|
| < 20 | Underutilized slots; consider consolidating operators or reducing KPUs |
| 20–40 | Healthy range for most workloads |
| 40–100 | Monitor GC pressure and checkpoint duration closely |
| 100–200 | Likely experiencing performance degradation; consider restructuring |
| > 200 | Split the job or restructure the graph (see Operator-to-Task-Slot Overload) |

### Using `disableChaining()` and `startNewChain()` Strategically

Break chains only when you have a specific reason:

```java
// Break the chain before a CPU-intensive operator to isolate its metrics
DataStream<Result> results = events
    .keyBy(Event::getKey)
    .process(new ExpensiveProcessor())
    .startNewChain()  // This operator starts a new chain
    .uid("expensive-processor-uid");

// Completely disable chaining for a specific operator (rarely needed)
DataStream<Enriched> enriched = events
    .map(new ExternalServiceLookup())
    .disableChaining()  // Runs in its own task, not chained with anything
    .uid("external-lookup-uid");
```

**When to break chains:**

- To isolate a CPU-intensive operator so its metrics (busyTime, backpressure) are visible independently in the Flink Web UI
- To separate an operator that makes external calls (HTTP, database) from the rest of the pipeline so its latency does not mask upstream/downstream metrics
- To control parallelism boundaries — operators with different parallelism cannot chain anyway, but `startNewChain()` makes the intent explicit

**When NOT to break chains:**

- For debugging only — use the Web UI's subtask metrics instead
- To "improve parallelism" — breaking chains does not change parallelism; it only adds serialization overhead
- Preemptively on every operator — this defeats the purpose of chaining and increases resource consumption

### Viewing the Physical Execution Plan

The Flink Web UI shows two views of the job:

1. **Job Graph** (Overview tab): Shows the logical plan with chained operators grouped into task boxes. Use this to verify chaining decisions.
2. **Task Managers** tab: Shows which task slots are allocated on each TaskManager and what tasks run in each slot. Use this to verify operator-to-task-slot distribution.

To check operator-to-task-slot assignments:

1. Open the Flink Web UI → select the running job
2. Click on a task box in the job graph to expand its details
3. The "Subtasks" tab shows each parallel instance (subtask) and which TaskManager hosts it
4. Cross-reference with the TaskManagers tab to see total slot utilization per TaskManager

For data skew detection and mitigation, the monolith job anti-pattern, and the high fan-out anti-pattern, see [job-graph-anti-patterns.md](job-graph-anti-patterns.md).

## Operator-to-Task-Slot Overload

### Recommended Ratios and Performance Implications

Each task slot runs a parallel pipeline of chained operators within a single thread. The more operators packed into a slot, the more work that thread must perform — including state access, serialization, timer management, and checkpoint barrier handling.

**Recommended ratio: 20–40 operators per task slot.**

| Ratio Range | Impact |
|-------------|--------|
| 20–40 | Optimal. Checkpoint barriers propagate quickly, GC pressure is manageable, per-operator metrics remain meaningful. |
| 40–100 | Elevated GC pressure from increased object allocation. Checkpoint duration starts to grow as more state must be snapshotted per slot. Latency percentiles widen. |
| 100–200 | Noticeable degradation. GC pauses become frequent, checkpoint durations may approach the checkpoint interval, and tail latency increases significantly. |
| > 200 | Critical. Split the job or restructure the graph. At this density, GC overhead dominates CPU time, checkpoints risk timing out, and individual operator metrics become unreliable. |

### Symptoms of Task Slot Overload

- **High GC pressure**: `heapMemoryUtilization` sustained above 80% (scale-up signal; see [monitoring-and-metrics.md](monitoring-and-metrics.md)), frequent full GC pauses
- **Slow checkpoints**: `lastCheckpointDuration` increasing or approaching the checkpoint interval
- **Increased latency**: `busyTimeMsPerSecond` approaching 1000 (fully saturated)
- **Unresponsive heartbeats**: In extreme cases, TaskManagers miss heartbeat deadlines causing restarts

### Threshold: >200 Operators → Split or Restructure

1. **Split the job** into independent Flink applications for processing paths that don't share state
2. **Restructure the graph**: combine sequential map/filter operations into a single `ProcessFunction`; remove redundant operators
3. **Adjust slot sharing groups** to distribute operators across more slots:

```java
DataStream<Result> results = events
    .keyBy(Event::getKey)
    .process(new HeavyProcessor())
    .slotSharingGroup("heavy-processing")
    .uid("heavy-processor-uid");
```

### Operator Group Scheduling

Flink's slot sharing allows operators from different pipeline parts to share the same task slot. When all operators are in the default group and the job has many operators, every slot runs one subtask of every operator — leading to overload.

**Strategies**: Group operators by resource profile (CPU-intensive vs I/O-bound in separate groups). Use the Flink Web UI's TaskManagers tab to verify balanced slot utilization.

## References

- See [Resource Optimization](resource-optimization.md) for KPU sizing, operator parallelism tuning, and checkpoint resource impact
- See [Best Practices](best-practices.md) for state management and serialization guidance
- See [Monitoring and Metrics](monitoring-and-metrics.md) for CloudWatch metric details and alarm configuration
- See [Managed Service for Apache Flink Overview](msf-overview.md) for KPU resource model and service constraints

# Kinesis Connector Guide

This guide covers Kinesis connector configuration for Amazon Managed Service for Apache Flink applications.

## Maven Dependencies

**CRITICAL:** Use the correct Kinesis connector dependency from the official Apache Flink project:

```xml
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-aws-kinesis-streams</artifactId>
    <version>${kinesis-streams.version}</version>
</dependency>
```

Where `kinesis-streams.version` should match your Flink version (e.g., `5.1.0-1.20` for Flink 1.20, `6.0.0-2.0` for Flink 2.2). See `dependency-management.md` for the full version mapping.

## Kinesis Source Configuration

When creating a Kinesis source, use the `KinesisStreamsSource` builder pattern. This API is the same for both Flink 1.20 and 2.2.

### Correct API Pattern

```java
import org.apache.flink.configuration.Configuration;
import org.apache.flink.connector.kinesis.source.config.KinesisSourceConfigOptions;

Map<String, Properties> applicationProperties = loadApplicationProperties(env);
Properties inputConfig = applicationProperties.get("input.kinesis.config");
Map<String, String> configMap = new HashMap<>();
inputConfig.forEach((k, v) -> configMap.put(k.toString(), v.toString()));
Configuration sourceConfig = Configuration.fromMap(configMap);
String inputStreamArn = inputConfig.getProperty("stream.arn");

KinesisStreamsSource<Event> source = KinesisStreamsSource.<Event>builder()
    .setStreamArn(inputStreamArn)
    .setDeserializationSchema(new EventDeserializationSchema())
    .setSourceConfig(sourceConfig)
    .build();

DataStream<Event> events = env
        .fromSource(source, 
                    WatermarkStrategy.<Event>forBoundedOutOfOrderness(Duration.ofSeconds(5))
                            .withTimestampAssigner((event, timestamp) -> event.getTimestamp())
                            .withIdleness(Duration.ofSeconds(10)),  // CRITICAL for low-throughput streams
                    "kinesis-source",
                    org.apache.flink.api.common.typeinfo.TypeInformation.of(Event.class))
        .name("kinesis-source")
        .uid("kinesis-source-uid")
        .filter(event -> event != null)
        .name("filter-null-events")
        .uid("filter-null-events-uid");
```

### Key Configuration Methods

- `.setStreamArn(String)` - Set the stream ARN (preferred over stream name)
- `.setDeserializationSchema(DeserializationSchema<T>)` - Set how to deserialize records
- `.setSourceConfig(Configuration)` - Set Kinesis client configuration
- `.build()` - Build the source

## Kinesis Sink Configuration

For Kinesis sinks, use the `KinesisStreamsSink` builder:

```java
Properties outputConfig = applicationProperties.get("output.kinesis.config");
String outputStreamArn = outputConfig.getProperty("stream.arn");

KinesisStreamsSink<Event> sink = KinesisStreamsSink.<Event>builder()
    .setStreamArn(outputStreamArn)
    .setSerializationSchema(new EventSerializationSchema())
    .setPartitionKeyGenerator(event -> String.valueOf(event.hashCode()))
    .setKinesisClientProperties(outputConfig)
    .build();

events.sinkTo(sink)
        .name("kinesis-sink")
        .uid("kinesis-sink-uid");
```

### Key Sink Methods

- `.setStreamArn(String)` - Set the destination stream ARN
- `.setSerializationSchema(SerializationSchema<T>)` - Set how to serialize records
- `.setPartitionKeyGenerator(PartitionKeyGenerator<T>)` - Set partition key logic
- `.build()` - Build the sink

## Polling Configuration and Throttling

The Kinesis `GetRecords` API has a hard limit of 5 calls per second per shard, shared across all consumers reading from that shard. The Flink Kinesis connector's default polling behavior can be aggressive and lead to `ReadProvisionedThroughputExceeded` or `LimitExceededException` errors, especially when multiple consumers share a stream or when polling intervals are too short.

**Key polling configuration options** (see [KinesisSourceConfigOptions Javadoc](https://www.javadoc.io/static/org.apache.flink/flink-connector-aws-kinesis-streams/6.0.0-2.0/org/apache/flink/connector/kinesis/source/config/KinesisSourceConfigOptions.html)):

| Config Option | Description | Default | Recommendation |
|---|---|---|---|
| `SHARD_GET_RECORDS_MAX` | Max records per `GetRecords` call | 10,000 (but Kinesis limit is 10,000 records / 10 MB per call; throttling occurs at 1,000 records for some stream configurations) | Lower to 1,000 or less if seeing throttling |
| `READER_EMPTY_RECORDS_FETCH_INTERVAL` | Interval between polling calls when no records are returned | 200ms (5 calls/sec) | Increase to 500ms–1s if sharing shards with other consumers or if the default rate exceeds the per-shard limit |
| `SHARD_DISCOVERY_INTERVAL` | Interval for discovering new shards via `ListShards` | 10s | Increase if `ListShards` rate limiting is observed |

**Tuning polling to avoid throttling:**

```java
Configuration sourceConfig = new Configuration();

// Reduce max records per GetRecords call to stay within Kinesis limits
sourceConfig.set(KinesisSourceConfigOptions.SHARD_GET_RECORDS_MAX, 1000);

// Increase polling interval to reduce GetRecords call rate
// Default is 200ms (5 calls/sec) — increase if sharing shards with other consumers
sourceConfig.set(KinesisSourceConfigOptions.READER_EMPTY_RECORDS_FETCH_INTERVAL, Duration.ofMillis(500));

KinesisStreamsSource<String> source = KinesisStreamsSource.<String>builder()
    .setStreamArn("arn:aws:kinesis:us-east-1:123456789012:stream/my-stream")
    .setDeserializationSchema(new SimpleStringSchema())
    .setSourceConfig(sourceConfig)
    .build();
```

**When polling tuning is sufficient (vs switching to EFO):**

- Single consumer reading from the stream: tune polling — EFO is unnecessary cost since there is no shared-quota contention.
- 2+ consumers and still seeing `ReadProvisionedThroughputExceeded` after raising `READER_EMPTY_RECORDS_FETCH_INTERVAL`: switch to EFO. Tuning polling intervals across multiple consumers is fragile; EFO eliminates the shared quota entirely.
- Polling interval increase introduces unacceptable latency: switch to EFO (HTTP/2 push has no polling delay).
- You expect to scale to more consumers later: prefer EFO upfront.

See [kinesis-efo-guide.md](kinesis-efo-guide.md) for EFO configuration, consumer lifecycle, and the full when-to-use-EFO checklist.

**Diagnosing polling throttling:**

- Check CloudWatch metric `ReadProvisionedThroughputExceeded` on the Kinesis stream — sustained values > 0 indicate throttling.
- Check Managed Service for Apache Flink CloudWatch logs for `LimitExceededException` errors.
- Monitor `GetRecords.Latency` and `GetRecords.Success` metrics to correlate throttling with read performance.

## Migrating from Legacy Kinesis Consumer to KinesisStreamsSource

**CRITICAL for Flink 2.x upgrades:** KDS connector versions below 5.0 have state that is incompatible with the Flink 2.2 connector (v6.0.0-2.0). You must migrate to connector v5.0+ on Flink 1.x before upgrading to Flink 2.x. See the [AWS blog post on the Kinesis source connector](https://aws.amazon.com/blogs/big-data/introducing-the-new-amazon-kinesis-source-connector-for-apache-flink/) for full details.

The legacy `FlinkKinesisConsumer` uses the removed `SourceFunction` interface and will not work with Flink 2.x. The `KinesisStreamsSource` uses the FLIP-27 Source API.

### Migration Paths

**DataStream API with operator UIDs defined:**

1. Update dependencies: replace `flink-connector-kinesis` with `flink-connector-aws-kinesis-streams` v5.0.0+
2. Replace `FlinkKinesisConsumer` with `KinesisStreamsSource` builder pattern
3. Change the UID of the source operator to a new string (this selectively resets source state while preserving all other operator state)
4. Configure starting position with `AT_TIMESTAMP` set to just before deployment time
5. Deploy with `allowNonRestoredState = true`

**Table API/SQL or DataStream without operator UIDs:**

1. Update dependencies and code as above
2. Deploy with `SKIP_RESTORE_FROM_SNAPSHOT` since Flink cannot map old operator state to new operators
3. After the application is running, switch back to `RESTORE_FROM_LATEST_SNAPSHOT` for future restarts

### Key Differences from Legacy Connector

| Feature | Legacy `FlinkKinesisConsumer` | New `KinesisStreamsSource` |
|---------|-------------------------------|---------------------------|
| Interface | `SourceFunction` (removed in 2.x) | FLIP-27 Source API |
| Stream identifier | Stream name | Stream ARN (cross-region/account support) |
| Watermarks | Implicit defaults | Explicit `WatermarkStrategy` required |
| Ordering on reshard | Not guaranteed | Guaranteed via parent-child shard lineage |
| Shard assigner | Even shard distribution | Uniform partition-key distribution |
| AWS SDK | v1 | v2 (non-blocking I/O) |
| JAR size | ~60 MB | ~200 KB |
| KCL/KPL dependency | Included | Removed (no built-in KPL de-aggregation) |

### State Compatibility Warning

The saved state from `FlinkKinesisConsumer` is not compatible with `KinesisStreamsSource`. You cannot restore source position from a snapshot taken with the legacy connector. Plan for either selective state reset (with UIDs) or full state reset (without UIDs) as described above.

## Authentication

In Managed Service for Apache Flink, authentication to Kinesis is handled automatically via the application's IAM execution role. No explicit credentials configuration is needed in the code.

# Enhanced Fan-Out (EFO) Configuration Guide

## Overview

This guide covers Enhanced Fan-Out (EFO) configuration for Kinesis sources in Managed Service for Apache Flink applications, including when to use EFO, source configuration, consumer lifecycle management, parallelism and shard count considerations, auto-scaling interaction, and troubleshooting.

For Kinesis source/sink configuration, authentication, and legacy consumer migration, see [kinesis-connector-guide.md](kinesis-connector-guide.md).

## When to Use EFO vs Standard Polling

Use Enhanced Fan-Out when:

- Multiple consumers read from the same Kinesis stream (EFO gives each consumer dedicated 2 MB/s per shard, vs shared 2 MB/s with polling)
- You need sub-200ms read latency (EFO uses HTTP/2 push, polling has up to 200ms delay per `GetRecords` call)
- Your application cannot tolerate `ReadProvisionedThroughputExceeded` throttling from competing consumers
- You are hitting the 5 `GetRecords` calls per second per shard limit due to multiple consumers or aggressive polling intervals

Standard polling is sufficient when:

- Your Flink application is the only consumer on the stream
- Latency requirements are relaxed (200ms+ acceptable)
- You want to minimize cost (EFO incurs additional per-consumer, per-shard-hour charges)

For polling configuration tuning, the 5 `GetRecords` calls/sec per shard limit, and diagnosing `ReadProvisionedThroughputExceeded`, see the "Polling Configuration and Throttling" section in [kinesis-connector-guide.md](kinesis-connector-guide.md). EFO is the right answer when polling tuning is not enough; the connector guide explains when to choose which.

## EFO Source Configuration

Enable EFO by setting `READER_TYPE` and `EFO_CONSUMER_NAME` in the source configuration:

```java
import org.apache.flink.connector.kinesis.source.KinesisStreamsSource;
import org.apache.flink.connector.kinesis.source.config.KinesisSourceConfigOptions;
import org.apache.flink.configuration.Configuration;

Configuration sourceConfig = new Configuration();
sourceConfig.set(KinesisSourceConfigOptions.READER_TYPE, KinesisSourceConfigOptions.ReaderType.EFO);
sourceConfig.set(KinesisSourceConfigOptions.EFO_CONSUMER_NAME, "my-flink-efo-consumer");

KinesisStreamsSource<String> source = KinesisStreamsSource.<String>builder()
    .setStreamArn("arn:aws:kinesis:us-east-1:123456789012:stream/my-stream")
    .setDeserializationSchema(new SimpleStringSchema())
    .setSourceConfig(sourceConfig)
    .build();

// Recommended: set parallelism >= shard count for optimal per-shard throughput isolation
env.fromSource(source,
        WatermarkStrategy.<String>forBoundedOutOfOrderness(Duration.ofSeconds(5))
            .withIdleness(Duration.ofSeconds(10)),
        "kinesis-efo-source")
    .setParallelism(shardCount)  // match or exceed shard count
    .uid("kinesis-efo-source-uid");
```

The consumer name must be unique per stream but can be reused across different streams. Reusing an existing consumer name on the same stream will terminate the previous subscription.

## Consumer Lifecycle Management

By default (`JOB_MANAGED`), the `KinesisStreamsSource` automatically registers the stream consumer on job start and deregisters on graceful stop. For environments where you want external control:

- `JOB_MANAGED` (default): Flink registers/deregisters the consumer automatically. Preferred for most Managed Service for Apache Flink applications.
- `SELF_MANAGED`: You register the consumer externally via AWS CLI (`aws kinesis register-stream-consumer`) or SDK, then provide the consumer ARN to the job. Use this when multiple jobs share a consumer or when you need to control the consumer lifecycle independently.

## Source Parallelism and Shard Count

Source parallelism should ideally match or exceed the Kinesis shard count when using EFO for optimal throughput. If parallelism is less than the shard count, some subtasks handle multiple shards — this still works but reduces the throughput benefit of EFO. If parallelism exceeds the shard count, idle subtasks will block watermark generation unless `withIdleness()` is configured on the `WatermarkStrategy`.

| Scenario | Parallelism vs Shards | Effect |
|---|---|---|
| Parallelism = shard count | 1:1 mapping | Optimal — each subtask gets dedicated 2 MB/s |
| Parallelism < shard count | Some subtasks handle multiple shards | Works but reduces per-shard throughput isolation |
| Parallelism > shard count | Idle subtasks | Requires `withIdleness()` or watermarks stall |

## EFO Interaction with Managed Service for Apache Flink Auto-Scaling and KPU Allocation

When Managed Service for Apache Flink auto-scaling adjusts KPU count, the total parallelism changes. This affects EFO consumers:

- Scaling up increases parallelism, potentially creating idle subtasks if parallelism exceeds shard count — ensure `withIdleness()` is set
- Scaling down reduces parallelism, causing subtasks to handle more shards — EFO still provides dedicated throughput per shard but each subtask processes more data
- During scaling events, the EFO consumer subscription is re-established automatically; expect brief transient errors in logs (this is normal)
- Set Managed Service for Apache Flink auto-scaling min KPU to ensure parallelism never drops below shard count for optimal EFO throughput

## Troubleshooting EFO

**Consumer registration errors (`LimitExceededException`)**:

- Kinesis limits concurrent consumer registrations to 5 in `CREATING` state per account. If you see this during job startup, the consumer will retry automatically. For persistent failures, check if other applications are registering consumers simultaneously.
- Maximum consumers per stream: 20 (Provisioned/On-demand Standard) or 50 (On-demand Advantage). Verify you haven't hit the limit with `aws kinesis list-stream-consumers`.

**Throughput exceptions (`SubscribeToShard` failures)**:

- Each EFO consumer gets dedicated 2 MB/s per shard. If you see throughput errors, verify the consumer is properly registered (`ACTIVE` status) using `aws kinesis describe-stream-consumer`.
- Transient `SubscribeToShard` errors are expected — subscriptions last 5 minutes and are automatically re-acquired.

**IAM permissions for EFO**:
The Managed Service for Apache Flink application's IAM execution role needs these additional permissions beyond standard Kinesis read access:

```json
{
    "Effect": "Allow",
    "Action": [
        "kinesis:RegisterStreamConsumer",
        "kinesis:DeregisterStreamConsumer",
        "kinesis:DescribeStreamConsumer",
        "kinesis:SubscribeToShard"
    ],
    "Resource": [
        "arn:aws:kinesis:us-east-1:123456789012:stream/my-stream",
        "arn:aws:kinesis:us-east-1:123456789012:stream/my-stream/consumer/*"
    ]
}
```

Note the consumer resource ARN (`stream/*/consumer/*`) — `SubscribeToShard` and `DescribeStreamConsumer` require permissions on the consumer resource, not just the stream.

**Retry strategy tuning**:
If `DescribeStreamConsumer` calls fail during startup (common when the consumer was just registered and is still in `CREATING` state), tune the dedicated retry strategy:

```java
sourceConfig.set(KinesisSourceConfigOptions.EFO_DESCRIBE_CONSUMER_RETRY_STRATEGY_MAX_ATTEMPTS, 30);
sourceConfig.set(KinesisSourceConfigOptions.EFO_DESCRIBE_CONSUMER_RETRY_STRATEGY_MIN_DELAY, Duration.ofSeconds(2));
sourceConfig.set(KinesisSourceConfigOptions.EFO_DESCRIBE_CONSUMER_RETRY_STRATEGY_MAX_DELAY, Duration.ofSeconds(10));
```

# Logging Configuration Guide

## Overview

This guide covers logging best practices for Amazon Managed Service for Apache Flink applications, including Managed Service for Apache Flink service-level vs. application-level logging, local development logging, production log level guidance, querying in CloudWatch Logs Insights, and rate-limited logging patterns for high-throughput operators. Use it when setting up logging for a new application, tuning log levels for production, or adding custom logging within Flink operators without flooding CloudWatch Logs.

### 1. Managed Service for Apache Flink Service-Level Monitoring Log Level

This is the log level configured through the Managed Service for Apache Flink API or console. It controls the application's log level.

- Set via the `MonitoringConfiguration.LogLevel` parameter in `CreateApplication` or `MonitoringConfigurationUpdate.LogLevelUpdate` in `UpdateApplication`.
- Also configurable in the console under the "Monitoring log level" section of the application configuration page.
- Valid values: `ERROR`, `WARN`, `INFO`, `DEBUG`.
- AWS recommends `INFO` because Apache Flink logs some errors at the INFO level rather than ERROR. Using ERROR alone may cause you to miss important failure information.
- `DEBUG` should only be used temporarily for troubleshooting — it significantly impacts application performance.

### 2. Application-Level Log4j2 Configuration

This is the Log4j2 configuration bundled inside your application JAR at `src/main/resources/log4j2.properties` (or `log4j2.xml`). **In Managed Service for Apache Flink, the service-level `MonitoringConfiguration.LogLevel` is the only log level control — it sets the application-wide log level. MSF does not support per-package or per-logger log level configuration through bundled Log4j2 config files.** Your bundled Log4j2 configuration is only used for local development. For per-package verbosity control in MSF, you can programmatically adjust log levels in your code using Log4j2's `Configurator.setLevel()` API, or use a separate DataStream with a sink (e.g., S3 or CloudWatch) for detailed debug output.

## Enabling CloudWatch Logging for an Managed Service for Apache Flink Application

To configure logging, you need:

1. **Create the CloudWatch log group AND a log stream inside it.** MSF does not auto-create either resource. If the log group exists but the log stream does not, MSF silently drops application logs — this is the most common cause of "logs configured but nothing shows up." Verify both with explicit CLI calls before changing log levels or redeploying:

   ```bash
   # Required: create the log group
   aws logs create-log-group --log-group-name /aws/kinesis-analytics/<app-name>

   # Required: create at least one log stream inside it (MSF needs this to exist)
   aws logs create-log-stream \
       --log-group-name /aws/kinesis-analytics/<app-name> \
       --log-stream-name kinesis-analytics-log-stream

   # Strongly recommended: set an explicit retention policy. CloudWatch defaults
   # to "never expire", which inflates storage cost over time. 30 days is a
   # reasonable production starting point; align with your audit requirements.
   aws logs put-retention-policy \
       --log-group-name /aws/kinesis-analytics/<app-name> \
       --retention-in-days 30
   ```

   Verify both resources exist:

   ```bash
   aws logs describe-log-groups --log-group-name-prefix /aws/kinesis-analytics/<app-name>
   aws logs describe-log-streams --log-group-name /aws/kinesis-analytics/<app-name>
   ```

2. Add the logging option to your application using `CreateApplication` (with `CloudWatchLoggingOptions`) or `AddApplicationCloudWatchLoggingOption` for an existing application. The `LogStreamARN` must point at the existing log stream from step 1, not just the log group.
3. Add the required IAM permissions to your application's service execution role:
   - `logs:PutLogEvents`
   - `logs:DescribeLogGroups`
   - `logs:DescribeLogStreams`

### Diagnosing "App is RUNNING but no logs in CloudWatch"

When an application is processing records but no logs appear, work this checklist in order — do **not** start by changing log level or redeploying, since the most common cause is a missing resource, not a verbosity issue.

1. **Confirm the logging option is attached.** Run `describe-application` and check `CloudWatchLoggingOptionDescriptions`. If empty or missing, that is the problem — re-run step 2 above.
2. **Confirm the log stream exists** (not just the log group). Run the `describe-log-streams` call above. A log group alone is not enough; without an existing stream, logs are silently dropped.
3. **Confirm IAM permissions on the execution role** include the three `logs:*` actions listed above. Missing permissions also cause silent log loss.
4. Only after the resource and permission checks pass should you investigate `MonitoringConfiguration.LogLevel` — and only then to confirm it is `INFO` (the AWS-recommended level), not as a fix for missing logs.

## Local Development Logging vs Managed Service for Apache Flink Logging

### Local Development (IDE / Docker / Flink Mini-Cluster)

When running locally, logging behaves differently from Managed Service for Apache Flink in several ways:

- Logs go to local stdout/stderr (your terminal or IDE console). There is no CloudWatch Logs integration.
- The Managed Service for Apache Flink service-level `MonitoringConfiguration` does not exist locally. Only your Log4j2 configuration applies.
- You can change log levels by editing `log4j2.properties` and restarting the application without redeploying a JAR.

### Managed Service for Apache Flink (Production / Staging)

- All stdout/stderr output from TaskManagers and JobManager is captured and forwarded to CloudWatch Logs automatically.
- The service-level monitoring log level is the only log level control. MSF does not apply bundled Log4j2 log level configurations — per-package or per-logger levels set in `log4j2.properties` are ignored.
- CloudWatch Logs charges $0.50/GB ingested and $0.03/GB stored per month — verbose logging has direct cost impact.
- The Managed Service for Apache Flink service-level log level can be changed via `UpdateApplication` without redeploying the JAR.
- For per-package verbosity control, programmatically adjust log levels in your code (e.g., `Configurator.setLevel("com.mypackage", Level.DEBUG)`), or use a separate DataStream with a sink for detailed debug output.

## Writing Custom Log Messages to CloudWatch Logs

You can write custom messages from your application code using either Log4j2 directly or SLF4J (which delegates to Log4j2 under the hood in Flink).

### Using SLF4J (Recommended)

SLF4J is the standard logging facade used throughout Flink's own codebase. Using it keeps your code consistent with Flink internals and avoids direct Log4j2 API coupling.

```java
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class MyFlinkJob {
    private static final Logger LOG = LoggerFactory.getLogger(MyFlinkJob.class);

    public static void main(String[] args) {
        LOG.info("Application starting with configuration: {}", config);
        // NOTE: If config contains user-controlled values, sanitize to prevent log injection:
        // String sanitized = config.toString().replaceAll("[\\r\\n]", "");
    }
}
```

No additional Maven dependencies are needed — SLF4J and the Log4j2 binding are included in the Flink runtime.

### Using Log4j2 Directly

If you prefer the Log4j2 API directly, add these dependencies to your `pom.xml`:

```xml
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-api</artifactId>
    <version>2.23.1</version>
</dependency>
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>2.23.1</version>
</dependency>
```

```java
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;

public class MyFlinkJob {
    private static final Logger LOG = LogManager.getLogger(MyFlinkJob.class);

    public static void main(String[] args) {
        LOG.info("This message will appear in CloudWatch Logs");
    }
}
```

In Managed Service for Apache Flink, custom log messages appear in CloudWatch Logs as structured JSON entries containing fields like `locationInformation`, `logger`, `message`, `threadName`, `applicationARN`, `applicationVersionId`, and `messageType`.

AWS recommends using the `INFO` level for custom messages because the application log contains a large volume of entries — INFO-level messages are easier to filter.

## Managed Service for Apache Flink-Specific CloudWatch Logs Insights Queries

These queries target the structured log format that Managed Service for Apache Flink produces natively (with `applicationARN`, `messageType`, etc.) and are useful for operational analysis beyond application-level logging.

```
# Distribution of tasks across TaskManagers (set time range to a single job run)
fields @timestamp, message
| filter message like /Deploying/
| parse message " to flink-taskmanager-*" as @tmid
| stats count(*) by @tmid
| sort @timestamp desc
| limit 2000

# Subtasks assigned to each TaskManager
fields @timestamp, @tmid, @subtask
| filter message like /Deploying/
| parse message "Deploying * to flink-taskmanager-*" as @subtask, @tmid
| sort @timestamp desc
| limit 2000

# Detect parallelism changes (auto-scaling or manual)
fields @timestamp, @parallelism
| filter message like /property: parallelism.default, /
| parse message "default, *" as @parallelism
| sort @timestamp asc

# Access denied errors
fields @timestamp, @message, @messageType
| filter @message like /AccessDenied/
| sort @timestamp desc

# Source or sink not found (Kinesis stream/resource missing)
fields @timestamp, @message
| filter @message like /ResourceNotFoundException/
| sort @timestamp desc

# Task failures — application switching from RUNNING to RESTARTING
fields @timestamp, @message
| filter @message like /switched from RUNNING to RESTARTING/
| sort @timestamp desc
```

# Monitoring and Metrics Guide

## Overview

This guide covers CloudWatch metrics analysis and custom metrics emission for Amazon Managed Service for Apache Flink applications. Use it when setting up monitoring for a new application, interpreting metric behavior on an existing one, or adding application-specific business metrics via the Flink MetricGroup API.

## Key CloudWatch Metrics for Managed Service for Apache Flink

### Core Metrics Reference

Managed Service for Apache Flink publishes metrics to CloudWatch under the `AWS/KinesisAnalytics` namespace. The following table lists the key metrics, what they measure, and expected healthy ranges for a well-tuned application.

| Metric | Unit | Description | Level | Healthy Range | Notes |
|--------|------|-------------|-------|---------------|-------|
| `cpuUtilization` | Percentage | Percentage of CPU used per TaskManager JVM process. Managed Service for Apache Flink publishes one sample per TaskManager per reporting interval. | Application | 20–70% (per TM) | Use `Maximum` for hottest TaskManager, `Average` for mean across all. Only accounts for TaskManager JVM CPU, not other container processes. Sustained > 70% indicates CPU-bound; consider adding KPUs. |
| `containerCPUUtilization` | Percentage | CPU utilization across task manager containers. Publishes samples per TaskManager per minute. | Application | 20–75% (per container) | More complete than `cpuUtilization` — includes all processes in the container, not just the JVM. Use this to detect CPU exhaustion at the container level. |
| `heapMemoryUtilization` | Percentage | JVM heap memory used per TaskManager. Managed Service for Apache Flink publishes one sample per TaskManager per reporting interval. | Application | 30–75% (per TM) | Use `Maximum` for hottest TaskManager, `Average` for mean. Only accounts for TaskManager JVM heap. Graduated thresholds: healthy ≤ 75%; sustained > 80% warrants investigation and scale-up; > 90% is critical (alarm). See [Recommended Alarms](#recommended-alarms-and-thresholds). |
| `containerMemoryUtilization` | Percentage | Memory utilization across task manager containers. Publishes 2× samples per TaskManager per minute. | Application | Below 85% | More complete than `heapMemoryUtilization` — includes working set memory. Better tracker of total memory exhaustion; upon exhaustion, results in OOM for the TaskManager pod. |
| `containerDiskUtilization` | Percentage | Disk utilization across task manager containers. Publishes 2× samples per TaskManager per minute. | Application | Below 80% | Represents utilization of the filesystem on which the container root volume is set up. |
| `oldGenerationGCCount` | Count | Total number of old-generation (full) GC operations across all task managers | Application | < 2 per minute | Frequent full GCs indicate heap pressure or memory leaks. Monitor `RATE(oldGenerationGCCount)` for trend. |
| `oldGenerationGCTime` | Milliseconds | Total time spent in old-generation GC operations | Application | < 500 ms per minute | High values cause latency spikes and checkpoint delays. AWS recommends alarming on `(oldGenerationGCTime * 100) / 60000` as a percentage of wall-clock time. |
| `threadsCount` | Count | Total number of live JVM threads used by the application | Application | Stable, proportional to parallelism | Not the same as application parallelism. Sudden increases may indicate thread leaks from custom connectors. |
| `lastCheckpointDuration` | Milliseconds | Time to complete the most recent checkpoint | Application | < 50% of checkpoint interval | Approaching the interval signals checkpoint pressure. Monitor `RATE(lastCheckpointDuration)` for trends. |
| `lastCheckpointSize` | Bytes | Total size of the most recent checkpoint | Application | Stable or slowly growing | Rapid growth indicates unbounded state accumulation. Monitor `RATE(lastCheckpointSize)` for trends. |
| `numberOfFailedCheckpoints` | Count | Cumulative count of failed checkpoints | Application | 0 | Any non-zero value warrants investigation. AWS recommends monitoring `RATE(numberOfFailedCheckpoints)` to alarm on the gradient, not absolute values. |
| `downtime` | Milliseconds | Time the application has been in a failing/recovering state | Application | 0 | **Flink 2.2: Deprecated** — use `restartingTime`, `cancellingTime`, `failingTime` instead. Returns 0 for running jobs, -1 for completed jobs. Any other value means the job is not running. |
| `uptime` | Milliseconds | Time since the last successful restart | Application | Continuously increasing | **Flink 2.2: Deprecated** — use `runningTime` instead. Returns -1 for completed jobs. Resets indicate restarts; frequent resets signal instability. |
| `fullRestarts` | Count | Total number of full restarts since job submission | Application | Low and stable | **Flink 2.2: Removed** — use `numRestarts` instead. Does not measure fine-grained restarts. Restarts can occur during internal Managed Service for Apache Flink maintenance; higher than normal indicates a problem. |
| `KPUs` | Count | Total number of KPUs used by the application | Application | N/A | Receives one sample per billing period (one hour). Use `MAX` or `AVG` over a period of at least 1 hour. Includes the orchestration KPU. |

#### Throughput Metrics

These metrics are available at Application, Operator, Task, and Parallelism levels. They have a special reporting behavior: Managed Service for Apache Flink takes **4 metric snapshots per minute**. When using the `SUM` statistic over a period, you must divide by 4 to get the correct value (metric math: `m1/4`).

| Metric | Unit | Description | Level | Notes |
|--------|------|-------------|-------|-------|
| `numRecordsIn` | Count | Total number of records received | Application, Operator, Task, Parallelism | When using `SUM` statistic, apply metric math `m1/4` because Managed Service for Apache Flink takes 4 snapshots per minute. |
| `numRecordsInPerSecond` | Count/Second | Records received per second | Application, Operator, Task, Parallelism | Same 4-snapshot-per-minute behavior — use `m1/4` with `SUM`. Healthy when matching expected input rate; drops indicate source issues or backpressure. |
| `numRecordsOut` | Count | Total number of records emitted | Application, Operator, Task, Parallelism | Same `m1/4` math applies with `SUM`. |
| `numRecordsOutPerSecond` | Count/Second | Records emitted per second | Application, Operator, Task, Parallelism | Same `m1/4` math applies with `SUM`. AWS recommends alarming when this falls below a minimum threshold. |
| `numLateRecordsDropped` | Count | Records dropped due to arriving late | Application, Operator, Task, Parallelism | Same `m1/4` math applies with `SUM`. |

**Important**: Select the metric at the correct Level. If you're tracking the metric for an Operator, you need to select the corresponding operator-level metric, not the application-level one.

#### Operator Performance Metrics

These metrics are available at Task, Operator, and Parallelism levels. They are available for Flink version 1.13 and later.

| Metric | Unit | Description | Level | Healthy Range | Notes |
|--------|------|-------------|-------|---------------|-------|
| `backPressuredTimeMsPerSecond` | Milliseconds | Time an operator spends backpressured per second | Task, Operator, Parallelism | < 100 ms/s | Sustained > 100 ms/s means a downstream operator cannot keep up. Available for Flink 1.13+. |
| `busyTimeMsPerSecond` | Milliseconds | Time an operator spends processing records per second | Task, Operator, Parallelism | < 800 ms/s | Approaching 1000 means the operator is fully saturated. Can be NaN if the value could not be calculated. Available for Flink 1.13+. |
| `idleTimeMsPerSecond` | Milliseconds | Time an operator is idle (no data to process) per second | Task, Operator, Parallelism | Varies by workload | Idle time excludes backpressured time — if the task is backpressured it is not idle. Available for Flink 1.13+. |

#### Watermark Metrics

| Metric | Unit | Description | Level | Notes |
|--------|------|-------------|-------|-------|
| `currentInputWatermark` | Milliseconds | The last watermark this application/operator/task/thread has received (epoch ms) | Application, Operator, Task, Parallelism | Large lag from wall-clock time indicates late data or watermark misconfiguration. Only emitted for dimensions with two inputs; represents the minimum of the last received watermarks. |
| `currentOutputWatermark` | Milliseconds | The last watermark this application/operator/task/thread has emitted (epoch ms) | Application, Operator, Task, Parallelism | AWS recommends monitoring `currentOutputWatermark - currentInputWatermark` to detect watermark drift. |

#### Managed Memory Metrics (Flink 1.13+)

These metrics relate to memory managed by Flink outside the Java heap, used for the RocksDB state backend.

| Metric | Unit | Description | Level | Notes |
|--------|------|-------------|-------|-------|
| `managedMemoryUsed` | Bytes | Amount of managed memory currently used | Application, Operator, Task, Parallelism | Available for Flink 1.13+. |
| `managedMemoryTotal` | Bytes | Total amount of managed memory | Application, Operator, Task, Parallelism | Available for Flink 1.13+. |
| `managedMemoryUtilization` | Percentage | Derived by `managedMemoryUsed / managedMemoryTotal` | Application, Operator, Task, Parallelism | Available for Flink 1.13+. Important for RocksDB state backend users. |

#### Kinesis Source Metrics

| Metric | Unit | Description | Level | Notes |
|--------|------|-------------|-------|-------|
| `millisBehindLatest` | Milliseconds | Consumer lag — how far behind the head of the stream the consumer is | Application (for Stream), Parallelism (for ShardId) | A value of 0 means caught up. A value of -1 means not yet reported. Growing lag means the application cannot keep up with input rate. Healthy: < 60,000 (1 min). |
| `bytesRequestedPerFetch` | Bytes | Bytes requested in a single call to `getRecords` | Application (for Stream), Parallelism (for ShardId) | **Flink 2.2: Removed** in KDS connector v6.0.0-2.0. Only available with legacy connector versions. |

#### Kafka Source Metrics

| Metric | Unit | Description | Level | Notes |
|--------|------|-------------|-------|-------|
| `records_lag_max` | Count | Maximum lag in number of records for any partition in this window | Application, Operator, Task, Parallelism | Use this for Kafka sources the same way `millisBehindLatest` is used for Kinesis sources. |
| `currentoffsets` | N/A | Consumer's current read offset, per partition | Application (for Topic), Parallelism (for PartitionId) | |
| `committedoffsets` | N/A | Last successfully committed offsets to Kafka, per partition | Application (for Topic), Parallelism (for PartitionId) | |
| `commitsFailed` | N/A | Total number of offset commit failures to Kafka | Application, Operator, Task, Parallelism | Commit failure does not affect integrity of Flink's checkpointed partition offsets. |
| `commitsSucceeded` | N/A | Total number of successful offset commits to Kafka | Application, Operator, Task, Parallelism | |
| `bytes_consumed_rate` | Bytes | Average number of bytes consumed per second for a topic | Application, Operator, Task, Parallelism | |

### How cpuUtilization and heapMemoryUtilization Are Reported

A common source of confusion: these metrics are **not** reported as a single summed value across all TaskManagers. Instead, Managed Service for Apache Flink publishes **one sample per TaskManager** per reporting interval.

With 5 TaskManagers, CloudWatch receives 5 data points per interval. CloudWatch statistics then work naturally:

- `Average` = mean utilization across all TaskManagers
- `Maximum` = the hottest (most utilized) TaskManager
- `Minimum` = the least utilized TaskManager

**You do not need to divide by KPU count.** Use `Maximum` to find the bottleneck TaskManager, or `Average` for overall health. The AWS docs recommend using `Maximum` for alarm thresholds.

**Note**: `containerCPUUtilization` and `containerMemoryUtilization` publish **2× the number of samples** per TaskManager per minute (2 × number of TaskManagers).

### Metric Dimensions

Managed Service for Apache Flink metrics are available at four dimension levels:

| Dimension | Granularity | When to Use |
|-----------|-------------|-------------|
| **Application** | Entire Managed Service for Apache Flink application | Default for dashboards and alarms. Provides a single view of overall application health. Use for `cpuUtilization`, `heapMemoryUtilization`, `downtime`, `uptime`, checkpoint metrics, and container metrics. |
| **Task** | Per-task (group of chained operators) | Use when diagnosing which part of the pipeline is the bottleneck. `backPressuredTimeMsPerSecond` and `busyTimeMsPerSecond` at the task level reveal which operator chain is overloaded. |
| **Operator** | Per-operator (individual operator within a task) | Use for fine-grained debugging of specific operators. `numRecordsInPerSecond` and `numRecordsOutPerSecond` at the operator level help identify data skew or filtering ratios. |
| **Parallelism** | Per-subtask (individual parallel instance) | Most granular level. Use for diagnosing data skew across subtasks. Kinesis `millisBehindLatest` uses this for per-ShardId metrics. Kafka offsets use this for per-PartitionId metrics. |

**Dimension selection guidance:**

- Start with Application-level metrics for dashboards and alarms — they give the overall health picture with minimal cardinality.
- Drop to Task-level when Application-level metrics show a problem (e.g., high backpressure) and you need to identify which task chain is responsible.
- Use Operator-level for targeted debugging of specific operators.
- Use Parallelism-level only for diagnosing data skew or per-shard/partition analysis, not for persistent dashboards.

### Warning: Task-Level Metrics and High Cardinality

Task-level, operator-level, and parallelism-level metrics produce one CloudWatch metric per subtask per metric name. The total metric count is:

```
metric_count = num_metrics × total_parallelism × num_operators
```

With high parallelism and ParallelismPerKPU > 1, this can produce thousands of individual CloudWatch metrics, leading to:

- **High CloudWatch costs**: CloudWatch charges per metric per month. An application with Parallelism = 64 (32 KPUs × ParallelismPerKPU = 2) × 50 operator-level metrics = 3,200 custom metrics.
- **Dashboard performance degradation**: CloudWatch dashboards with thousands of metrics load slowly and become difficult to navigate.
- **Alarm limits**: CloudWatch has per-account alarm limits. Task-level alarms at high parallelism can consume a significant portion of the quota.

**Recommendation**: Use Application-level dimensions for all persistent alarms and dashboards. Reserve Task-level, Operator-level, and Parallelism-level dimensions for temporary diagnostic queries using CloudWatch Metrics Insights or the Flink Web UI, which provides subtask-level metrics without CloudWatch cardinality costs.

## Recommended Alarms and Thresholds

### Critical Metric Alarms

Set up CloudWatch alarms for these metrics on every Managed Service for Apache Flink production application. All alarms should use the Application dimension unless otherwise noted. These align with the [AWS recommended alarms](https://docs.aws.amazon.com/managed-flink/latest/java/monitoring-metrics-alarms.html).

| Metric | Statistic | Period | Threshold | Alarm Condition | Severity |
|--------|-----------|--------|-----------|-----------------|----------|
| `downtime` (Flink 1.x) or `restartingTime` / `cancellingTime` / `failingTime` (Flink 2.2) | Average | 1 minute | > 0 | Any non-zero value means the job is not running. Use "1 out of 1" evaluation — any downtime is immediately actionable. | Critical |
| `RATE(numberOfFailedCheckpoints)` | Average | 5 minutes | > 0 | Monitor the rate (gradient), not absolute values. A non-zero rate means checkpoints are actively failing. | High |
| `Operator.numRecordsOutPerSecond` | Average | 5 minutes | < minimum expected throughput | Falling below this threshold indicates the application isn't making expected progress on input data. | High |
| `lastCheckpointDuration` | Maximum | 5 minutes | > 50% of checkpoint interval | Checkpoint taking too long; risk of queuing and timeout. Also monitor `RATE(lastCheckpointDuration)` for trends. | High |
| `lastCheckpointSize` | Maximum | 5 minutes | > maximum expected size | Continuously increasing size indicates unbounded state accumulation. Also monitor `RATE(lastCheckpointSize)`. | High |
| Memory utilization (`heapMemoryUtilization` and `containerMemoryUtilization`) | Maximum | 5 minutes | heap > 90% / container > 85% |  **`heapMemoryUtilization`** is what AWS officially recommends in [Use CloudWatch Alarms with Amazon Managed Service for Apache Flink](https://docs.aws.amazon.com/managed-flink/latest/java/monitoring-metrics-alarms.html); threshold 90% Maximum, "3 out of 3 consecutive periods". Sees only the TaskManager JVM heap. **`containerMemoryUtilization`** catches container-level OOMs that heap misses (off-heap RocksDB state, native libraries, JVM overhead) — per the AWS [Metrics and dimensions](https://docs.aws.amazon.com/managed-flink/latest/java/metrics-dimensions.html) page, this is "a better tracker of total memory exhaustion." Use container as your primary memory alarm if your application is stateful with RocksDB. Either alone is acceptable; both is recommended for stateful apps. | High |
| `cpuUtilization` | Maximum | 5 minutes | > 80% | AWS recommends 80% threshold using `Maximum` statistic. Use "3 out of 3 consecutive periods" evaluation. | High |
| `records_lag_max` or `millisBehindLatest` | Maximum | 5 minutes | > maximum expected latency | Use `records_lag_max` for Kafka sources, `millisBehindLatest` for Kinesis sources. Rising above threshold means the application is falling behind. | Medium |
| `backPressuredTimeMsPerSecond` | Maximum | 5 minutes | > 500 ms/s sustained for 3 consecutive periods | Severe backpressure; downstream bottleneck. | High |
| `uptime` (Flink 1.x) or `runningTime` (Flink 2.2) | Minimum | 5 minutes | Resets (drops to 0) more than 2× in 1 hour | Frequent restarts indicate instability. | High |
| `threadsCount` | Maximum | 5 minutes | > maximum expected thread count | Watch for thread leaks in task managers. | Medium |
| `(oldGenerationGCTime * 100) / 60000` | Maximum | 1 minute | > threshold (set so typical GC is 60% of threshold) | AWS recommends this percentage-of-time formula. Continuously increasing indicates a memory leak. | Medium |
| `RATE(oldGenerationGCCount)` | Maximum | 5 minutes | > maximum expected rate | Continuously increasing indicates a memory leak. | Medium |
| `currentOutputWatermark - currentInputWatermark` | Minimum | 5 minutes | > threshold | Continuously increasing indicates the application is processing increasingly older events or an upstream subtask has stalled watermark emission. | Medium |

**Alarm configuration notes:**

- Use "3 out of 3 consecutive periods" evaluation for CPU, heap, and backpressure alarms to avoid false alarms from transient spikes during checkpoints or GC.
- For `downtime`, use "1 out of 1" — any downtime is immediately actionable.
- For `lastCheckpointDuration`, calculate the threshold from your configured checkpoint interval. If the interval is 60 seconds, alarm at > 30,000 ms.
- For `millisBehindLatest` / `records_lag_max`, combine the threshold with a trend check: a one-time spike that recovers is less concerning than a steadily increasing value.

### Using Metric Trends to Drive Scaling Decisions

Point-in-time metric values can be misleading. Use trends over 30–60 minute windows to make scaling decisions:

**Scale up (add KPUs) when:**

- `cpuUtilization` `Maximum` is sustained above 70% for 30+ minutes during normal (non-peak) traffic. This leaves insufficient headroom for traffic spikes and checkpoint overhead.
- `backPressuredTimeMsPerSecond` is sustained above 100 ms/s for 15+ minutes. This means at least one operator chain cannot keep up with its input rate.
- `millisBehindLatest` or `records_lag_max` is monotonically increasing over 30+ minutes. The application is falling further behind and will not recover without more resources.
- `lastCheckpointDuration` is trending upward and approaching the checkpoint interval. State is growing faster than the application can snapshot it.

**Scale down (reduce KPUs) when:**

- `cpuUtilization` `Maximum` is sustained below 30% for 60+ minutes during peak traffic. The application is over-provisioned.
- `backPressuredTimeMsPerSecond` is consistently 0 for 60+ minutes. No operators are bottlenecked.
- `heapMemoryUtilization` `Maximum` is sustained below 40% for 60+ minutes. Memory is significantly over-provisioned.

**Do not scale based on:**

- Short spikes (< 10 minutes) in CPU or backpressure — these are often caused by checkpoint processing or transient traffic bursts and resolve on their own.
- A single high `lastCheckpointDuration` value — one slow checkpoint can be caused by a GC pause or S3 latency spike. Look for a trend of increasing duration across multiple checkpoints.
- `numRecordsInPerSecond` alone — throughput fluctuations are normal. Combine with backpressure and lag metrics to determine if the application is actually struggling.

## Custom Metrics Emission

### Registering Custom Metrics via Flink MetricGroup API

Flink provides four metric types through the `MetricGroup` API, available in any `RichFunction` or `ProcessFunction` via `getRuntimeContext().getMetricGroup()`:

| Metric Type | Purpose | Registration Method | Example Use Case |
|-------------|---------|-------------------|------------------|
| **Counter** | Monotonically increasing count | `metricGroup.counter("name")` | Records processed, errors encountered, records filtered |
| **Gauge** | Point-in-time value that can go up or down | `metricGroup.gauge("name", gaugeFunction)` | Queue depth, cache size, current processing delay |
| **Meter** | Rate of events (events per second) | `metricGroup.meter("name", new MeterView(60))` | Throughput rate, error rate over a sliding window |
| **Histogram** | Distribution of values (min, max, mean, percentiles) | `metricGroup.histogram("name", new DescriptiveStatisticsHistogram(100))` | Record processing latency, payload sizes |

### Critical: The `kinesisanalytics` Metric Group Requirement

**Managed Service for Apache Flink publishes only metrics registered under the `kinesisanalytics` metric group to CloudWatch.** If you register custom metrics directly on `getRuntimeContext().getMetricGroup()` without adding the `kinesisanalytics` group, your metrics will be visible in the Flink Web UI but will **not** appear in CloudWatch.

You must use `.addGroup("kinesisanalytics")` when registering custom metrics:

```java
// CORRECT — metrics will appear in CloudWatch
getRuntimeContext().getMetricGroup()
    .addGroup("kinesisanalytics")
    .counter("myCounter");

// WRONG — metrics will NOT appear in CloudWatch
getRuntimeContext().getMetricGroup()
    .counter("myCounter");
```

The `kinesisanalytics` group name is a legacy from when the service was called Kinesis Data Analytics. Additional metric groups can be included and will be published to CloudWatch as dimensions.

### How Custom Metrics Publish to CloudWatch from Managed Service for Apache Flink

Managed Service for Apache Flink automatically publishes all Flink metrics registered under the `kinesisanalytics` group to CloudWatch under the `AWS/KinesisAnalytics` namespace. Custom metrics appear in CloudWatch with the same dimensions as built-in metrics (Application, Task, Operator, Parallelism).

**Naming conventions:**

- Use descriptive concatenated names: `ReceivedRecordsTotal`, `FilteredRecordsAverage` (as shown in the AWS examples)
- Prefix with a domain to avoid collisions with Flink internal metrics
- Keep names short — they contribute to CloudWatch metric cardinality and appear in dashboards

### Code Example

The following example demonstrates the pattern from the [official AWS CustomMetrics sample](https://github.com/aws-samples/amazon-managed-service-for-apache-flink-examples/tree/main/java/CustomMetrics). It uses a `RichMapFunction` as a pass-through that emits a counter and a gauge:

```java
public class MetricEmittingMapperFunction extends RichMapFunction<SpeedRecord, SpeedRecord> {
    private static final double AVERAGE_DECAY = 0.1;

    private transient Counter counter;
    private transient double runningAverage;
    private final String customMetricName;

    public MetricEmittingMapperFunction(final String customMetricName) {
        this.customMetricName = customMetricName;
    }

    @Override
    public void open(OpenContext openContext) throws Exception {
        // CRITICAL: Must use .addGroup("kinesisanalytics") for CloudWatch publishing
        this.counter = getRuntimeContext().getMetricGroup()
                .addGroup("kinesisanalytics")
                .counter(customMetricName + "Total");

        getRuntimeContext().getMetricGroup()
                .addGroup("kinesisanalytics")
                .gauge(customMetricName + "Average", () -> runningAverage);
    }

    @Override
    public SpeedRecord map(SpeedRecord value) {
        counter.inc();
        runningAverage = runningAverage * (1 - AVERAGE_DECAY) +
                value.speed * AVERAGE_DECAY;
        return value;
    }
}
```

This mapper is then used in the pipeline to emit metrics at different stages:

```java
// Emit metrics before filtering
DataStream<SpeedRecord> beforeFilter = input
        .map(new MetricEmittingMapperFunction("ReceivedRecords"));

// Filter
DataStream<SpeedRecord> filtered = beforeFilter
        .filter(SpeedLimitFilter::isAboveSpeedLimit);

// Emit metrics after filtering
DataStream<SpeedRecord> afterFilter = filtered
        .map(new MetricEmittingMapperFunction("FilteredRecords"));
```

### Cardinality Limits and Management Strategies

CloudWatch has practical limits on custom metric cardinality that affect cost and usability:

- **CloudWatch pricing**: Each unique metric (combination of namespace, metric name, and dimension values) is billed as a custom metric. At $0.30 per metric per month, an application with 100 custom metrics × 32 subtasks = 3,200 billable metrics = ~$960/month.
- **Dashboard limits**: CloudWatch dashboards become slow and unwieldy with more than a few hundred metrics. Operator-level custom metrics at high parallelism quickly exceed this.
- **API throttling**: CloudWatch `PutMetricData` has API rate limits. Excessive custom metrics can cause metric publishing delays or dropped data points.

**Strategies for managing cardinality:**

1. **Register metrics at the operator level, not per-key.** Do not create a separate counter for each key value (e.g., `orders.processed.customerA`, `orders.processed.customerB`). Instead, use a single counter per operator and rely on subtask-level dimensions for distribution analysis.

2. **Use Application-level aggregation for alarms.** CloudWatch automatically aggregates subtask-level metrics to the Application level using the `Average`, `Sum`, `Min`, and `Max` statistics. Set alarms on Application-level aggregates rather than individual subtask metrics.

3. **Limit custom metrics to 10–20 per operator.** Beyond this, the cardinality cost grows quickly with parallelism. Focus on metrics that directly inform operational decisions (error rates, processing latency, business KPIs).

4. **Prefer counters and gauges over histograms.** Histograms publish multiple CloudWatch metrics per registration (min, max, mean, p50, p75, p95, p99, count) — a single histogram registration produces 8+ CloudWatch metrics per subtask.

### Metric Reporting Interval and Cost Implications

Managed Service for Apache Flink reports metrics to CloudWatch at a fixed interval, typically every 60 seconds (with 4 snapshots per minute for throughput metrics). This has several implications:

- **Granularity**: Custom metrics have 1-minute resolution in CloudWatch. Sub-minute fluctuations are averaged within each reporting interval. For latency-sensitive monitoring, use the Flink Web UI (which updates in near-real-time) alongside CloudWatch.
- **Cost**: CloudWatch charges are per-metric per-month, not per-data-point. The reporting interval does not directly affect cost — but more frequent reporting (if configurable in future Managed Service for Apache Flink versions) would not increase cost for the same metric set.
- **Meter accuracy**: Flink's `MeterView` calculates rates over a configurable window (e.g., 60 seconds). Align the meter window with the reporting interval for consistent rate values in CloudWatch. A 60-second `MeterView` window with 60-second reporting produces stable rate metrics.
- **Gauge staleness**: Gauges report their current value at each reporting interval. If the gauge value changes rapidly between intervals, CloudWatch only captures the value at the reporting moment. For rapidly changing values, consider using a counter (cumulative) and computing rates in CloudWatch metric math.

## Monitoring Setup Operations

Operational specifics for wiring up monitoring on a new or existing MSF application.

### Log Group / Stream Creation Order

The log group and stream **must exist before** `add-application-cloud-watch-logging-option` is called — MSF does not create them. Application logs are silently dropped if the destination doesn't exist.

```
1. aws logs create-log-group    --log-group-name /aws/kinesis-analytics/<app>
2. aws logs create-log-stream   --log-group-name /aws/kinesis-analytics/<app> \
                                --log-stream-name kinesis-analytics-log-stream
3. aws logs put-retention-policy --log-group-name /aws/kinesis-analytics/<app> \
                                 --retention-in-days 30
4. aws kinesisanalyticsv2 add-application-cloud-watch-logging-option ...
```

Default CloudWatch log retention is unlimited — set retention explicitly to control cost. 30 days is a reasonable default; 7 days for dev.

### MetricsLevel — Cost vs Granularity

| Level | Per-app metrics emitted | When to use |
|-------|------------------------|-------------|
| APPLICATION | ~25 metrics × 1 dimension | Production default |
| OPERATOR | ~25 × N operators | Diagnosing a specific bottleneck operator |
| TASK | ~25 × M tasks | Rare — usually OPERATOR is sufficient |
| PARALLELISM | ~25 × Parallelism × N operators | **Avoid above Parallelism=64** (CloudWatch metric explosion) |

OPERATOR/TASK/PARALLELISM are useful for short-window diagnosis; switch back to APPLICATION after. CloudWatch custom metrics are billed at ~$0.30/metric-month.

### `treat-missing-data` Semantics

The `--treat-missing-data` flag determines alarm state when a metric stops reporting. The right value differs per alarm:

| Metric | `treat-missing-data` | Why |
|--------|---------------------|-----|
| `downtime` | `breaching` | Missing data = app not reporting = a problem |
| `fullRestarts` / `numRestarts` (cumulative) | `notBreaching` | Counter only emits when non-zero in some Flink versions |
| `numberOfFailedCheckpoints` | `notBreaching` | Cumulative counter; resets on restart |
| `cpuUtilization`, `containerCPUUtilization`, `containerMemoryUtilization`, `heapMemoryUtilization` | `missing` (default) | Brief gaps during restart are normal |
| `backPressuredTimeMsPerSecond` | `missing` | Operator-level metric, gaps are normal |

**Cumulative counters need special treatment.** `numberOfFailedCheckpoints`, `fullRestarts`, and `numRestarts` only ever increase during a job's lifetime; they reset only when the application restarts. Three rules follow from this and must be configured together — getting any one wrong breaks the alarm.

1. **Alarm on `RATE()`, not the raw value.** `RATE(numberOfFailedCheckpoints) > 0` alarms when checkpoints are *actively* failing in the current window. An alarm on the raw cumulative value would trip on the first failure ever and stay in ALARM until the next application restart, even after the underlying issue was fixed, because the count never decreases.

2. **Set `treat-missing-data: notBreaching`.** Some Flink versions only emit these counters when the value is non-zero, so a healthy app produces emission gaps as a normal steady state. `notBreaching` makes those gaps count as "not in alarm" instead of flipping the alarm to INSUFFICIENT_DATA. Combined with rule 1, the lifecycle is: healthy app → no data points → OK; failures land → non-zero `RATE()` → ALARM; failures stop → gaps return → OK.

3. **Route the alarm as an investigation trigger, not a self-clearing health signal.** Rule 1 means the alarm *will* return to OK on its own once new failures stop landing in the window. That is a statement about *recent activity*, not about *resolution*: "no new failures in the last N minutes" is not the same as "an operator diagnosed the root cause." Send these alarms to a ticketing or acknowledgment workflow where a human closes them after investigation, not to a paging path that treats the OK transition as "problem solved." A burst of checkpoint failures that stops on its own is still worth a postmortem.

### Alarm Priority Order for New Applications

When setting up monitoring on a new app, configure in this order — earlier alarms catch issues that downstream alarms can't:

1. `RATE(numberOfFailedCheckpoints) > 0` — checkpoint failures = data loss risk
2. `downtime > 0` (Flink 1.x) or `restartingTime > 0` (Flink 2.2) — app not processing
3. `RATE(fullRestarts) > 0` (Flink 1.x) or `RATE(numRestarts) > 0` (Flink 2.2) — instability
4. `cpuUtilization > 80%` Maximum, 3/3 — capacity warning
5. `heapMemoryUtilization > 90%` Maximum, 3/3 — OOM risk (AWS's officially recommended alarm). Substitute or supplement with `containerMemoryUtilization > 85%` for stateful apps where off-heap RocksDB / native memory matters; AWS's metric docs call container "a better tracker of total memory exhaustion."
6. `backPressuredTimeMsPerSecond > 500` Average, 3/3 — bottleneck
7. `millisBehindLatest` (Kinesis) or `records_lag_max` (Kafka) — falling behind input

For exact thresholds and statistic recommendations, see [Recommended Alarms and Thresholds](#recommended-alarms-and-thresholds) above.

### Common Setup Mistakes

| Mistake | Consequence |
|---------|-------------|
| Adding logging option before creating log group | Logs silently dropped; no error |
| `treat-missing-data: missing` on `downtime` | Missed outages — a stopped app emits no `downtime` data |
| Alarming on cumulative counter without `RATE()` | One-time alert that never auto-resolves |
| Leaving MetricsLevel=PARALLELISM in production | CloudWatch metric cost explosion at scale |
| No log retention policy | Unbounded log storage cost |
| APPLICATION level when diagnosing a specific operator | Cannot isolate the bottleneck — temporarily switch to OPERATOR |

# MSF Constraints and Common Patterns

## Overview

This guide covers Managed Service for Apache Flink vs self-managed Flink differences, MSF-specific constraints (resource, network, storage limits), and common MSF patterns including streaming ETL and real-time analytics SQL examples.

For the MSF architecture overview, KPU resource model, and AWS service integration, see [msf-overview.md](msf-overview.md).

## Managed Service for Apache Flink vs Self-Managed Flink

### Key Differences

| Aspect | Managed Service for Apache Flink | Self-Managed Flink |
|--------|-----|-------------------|
| **Flink Version** | Flink 1.20 and 2.2 (managed updates) | Any version (manual upgrades) |
| **Infrastructure Management** | Fully managed by AWS | Manual cluster setup and maintenance |
| **Resource Model** | KPU-based scaling (1 vCPU, 4GB per KPU) | Manual instance type selection and scaling |
| **Parallelism Configuration** | Service-level through Managed Service for Apache Flink console | Application-level in code |
| **Checkpoint Configuration** | Service-level managed by Managed Service for Apache Flink | Application-level configuration required |
| **Savepoint Management** | Managed Service for Apache Flink console and API-based | Manual CLI or API operations |
| **Monitoring** | Integrated CloudWatch | Custom monitoring setup required |
| **Security** | Built-in IAM integration | Manual security configuration |
| **Cost Model** | Pay for KPUs used | Pay for entire cluster resources |

### Configuration Separation

**MSF-Managed (Service-Level)**: Checkpoint intervals/retention/storage, savepoint management, parallelism/KPU scaling, network/VPC/security groups, cluster configuration (JobManager/TaskManager), state backend (RocksDB), fault tolerance settings.

**User-Controlled (Application-Level)**: Business logic, custom serializers/data formats, application properties, source/sink connector configs, custom metrics, watermark strategies, UDFs.

### Advantages of Managed Service for Apache Flink

- No cluster management overhead; automatic failure recovery with service-level checkpoints
- Built-in CloudWatch monitoring and alerting; simplified deployment
- Native AWS service connectivity with optimized connectors; IAM-integrated security
- Automatic KPU-based scaling; elastic resource allocation with pay-per-use pricing

### Considerations for Migration

- Application code remains largely unchanged (target Flink 1.20 or 2.2)
- Remove application-level checkpoint/savepoint configuration (now MSF-managed)
- Update parallelism settings for KPU-based scaling; replace custom monitoring with CloudWatch
- Move checkpoint config to MSF service-level settings; remove cluster-level configurations

## Managed Service for Apache Flink-Specific Constraints and Optimizations

### Service Constraints

**Resource Limits**:

- Maximum parallelism per application: ParallelismPerKPU × KPU limit (default KPU limit is 64; request increase via Service Quotas)
- Maximum memory per KPU: 4 GB (1 vCPU, 4 GB memory, 50 GB storage per KPU)
- Maximum number of applications per account: 50 (adjustable through AWS support)
- Checkpoint interval minimum: 1 second (configured via Managed Service for Apache Flink console, not application code)
- Maximum KPU count per application: 250 (default quota is 64; request increase via Service Quotas)

**Network Constraints**:

- VPC-only deployment (no direct public internet access for security)
- Specific subnet requirements for high availability across multiple AZs
- Managed Service for Apache Flink-managed security group configuration for service communication
- NAT Gateway or VPC endpoints required for external AWS service connectivity
- Cross-region data transfer limitations for compliance and performance

**Storage Constraints**:

- Checkpoints automatically stored in Managed Service for Apache Flink-managed S3 buckets (user cannot configure location)
- Savepoints require user-specified S3 bucket in same region as Managed Service for Apache Flink application
- State backend: RocksDB by default (configurable via AWS support case)
- Recommended maximum state size per key: keep values small (low single-digit MB) for optimal RocksDB performance
- Checkpoint retention managed by Managed Service for Apache Flink service-level policies, not application configuration

## Common Managed Service for Apache Flink Patterns

### Streaming ETL Pattern

```java
KinesisStreamsSource<ProcessedRecord> kdsSource =
        KinesisStreamsSource.<ProcessedRecord>builder()
                .setStreamArn("arn:aws:kinesis:us-east-1:123456789012:stream/test-stream")
                .setSourceConfig(sourceConfig)
                .setDeserializationSchema(new ProcessedRecordDeserializationSchema())
                .setKinesisShardAssigner(ShardAssignerFactory.uniformShardAssigner())
                .build();
                
DataStream<ProcessedRecord> processed = env
    .fromSource(kdsSource,
            WatermarkStrategy.<ProcessedRecord>forBoundedOutOfOrderness(Duration.ofSeconds(5))
                    .withTimestampAssigner((event, ts) -> event.getTimestamp())
                    .withIdleness(Duration.ofSeconds(10)),
            "Kinesis Source")
    .keyBy(ProcessedRecord::getKey)
    .window(TumblingEventTimeWindows.of(Duration.ofMinutes(5)))
    .aggregate(new AggregationFunction());

KinesisStreamsSink<ProcessedRecord> kdsSink =
    KinesisStreamsSink.<ProcessedRecord>builder()
        .setKinesisClientProperties(sinkProperties)
        .setSerializationSchema(new ProcessedRecordSerializationSchema())
        .setPartitionKeyGenerator(element -> String.valueOf(element.hashCode()))
        .setStreamArn("arn:aws:kinesis:us-east-1:123456789012:stream/sink-stream")
        // IMPORTANT: true ensures the job fails on write errors, letting Flink's
        // checkpoint/restart mechanism retry rather than silently dropping records.
        // Use false only for best-effort delivery where availability is prioritized
        // over data completeness — but be aware that failed records are lost.
        .setFailOnError(true)
        .setMaxBatchSize(500)
        .setMaxInFlightRequests(50)
        .setMaxBufferedRequests(10_000)
        .setMaxBatchSizeInBytes(5 * 1024 * 1024)
        .setMaxTimeInBufferMS(5000)
        .setMaxRecordSizeInBytes(1 * 1024 * 1024)
        .build();

processed.sinkTo(kdsSink);
```

### Real-time Analytics Pattern

```sql
-- Managed Service for Apache Flink-optimized Flink SQL for real-time analytics
CREATE TABLE kinesis_source (
    user_id STRING,
    event_type STRING,
    timestamp_col TIMESTAMP(3),
    WATERMARK FOR timestamp_col AS timestamp_col - INTERVAL '5' SECOND
) WITH (
    'connector' = 'kinesis',
    'stream' = 'user-events',
    'aws.region' = 'us-east-1',
    'format' = 'json'
);

CREATE TABLE s3_sink (
    window_start TIMESTAMP(3),
    window_end TIMESTAMP(3),
    user_count BIGINT,
    event_count BIGINT
) WITH (
    'connector' = 'filesystem',
    'path' = 's3://analytics-bucket/results/',
    'format' = 'parquet'
);

INSERT INTO s3_sink
SELECT 
    window_start,
    window_end,
    COUNT(DISTINCT user_id) as user_count,
    COUNT(*) as event_count
FROM TABLE(
    TUMBLE(TABLE kinesis_source, DESCRIPTOR(timestamp_col), INTERVAL '1' HOUR))
GROUP BY window_start, window_end;
```

### Local Docker Configuration vs Managed Service for Apache Flink Service Configuration

**IMPORTANT DISTINCTION**: Local Docker configuration is for development only and differs significantly from Managed Service for Apache Flink service configuration.

#### Managed Service for Apache Flink Service Configuration (Production Deployment)

**Managed Service for Apache Flink KPU Configuration (Service Level)**
Configured through Managed Service for Apache Flink console - NOT in application code. KPU Configuration:

- Each KPU: 1 vCPU, 4 GB memory
- You configure `Parallelism` (total task slots) and `ParallelismPerKPU` (task slots per KPU, default 1, max 8)
- Allocated KPUs = Parallelism / ParallelismPerKPU
- Auto-scaling: adjusts `CurrentParallelism` within Min/Max KPU bounds
- Managed Service for Apache Flink automatically manages task slot allocation

**Managed Service for Apache Flink Service Configuration (Console/API Only)**
Managed Service for Apache Flink manages these through service configuration:

State backend:

- RocksDB (configurable via support case)
Checkpoint Configuration:
- Interval: 60 seconds (configurable)
- Storage: S3 (Managed Service for Apache Flink-managed bucket)
Savepoint Configuration:
- Storage: S3 (Managed Service for Apache Flink-managed)
- Triggered through Managed Service for Apache Flink console/API
Parallelism Configuration:
- Parallelism: Total task slots (service-level setting)
- ParallelismPerKPU: Task slots per KPU (default 1, max 8)
- Allocated KPUs = Parallelism / ParallelismPerKPU
- Auto-scaling: adjusts CurrentParallelism within configured bounds

# Managed Service for Apache Flink Overview and Concepts Guide

## Introduction

Amazon Managed Service for Apache Flink is a fully managed service that makes it easy to develop, deploy, and operate Apache Flink applications on AWS using Kiro IDE for development. Managed Service for Apache Flink supports Flink 1.20 and Flink 2.2. For new applications, default to Flink 2.2. For existing applications, use the user's current Flink version. This guide provides a comprehensive overview of Managed Service for Apache Flink architecture, KPU-based scaling, service-level configuration, and key differences from self-managed Flink deployments.

Managed Service for Apache Flink abstracts infrastructure complexity while providing the full power of Apache Flink for stream processing, with a clear separation between service-level configuration managed by Managed Service for Apache Flink and application-level configuration controlled by developers through Kiro IDE.

## Managed Service for Apache Flink Architecture Overview

### Service Architecture

Managed Service for Apache Flink abstracts away the complexity of managing Flink clusters while providing the full power of Apache Flink for stream processing. The service architecture consists of:

**Control Plane**:

- Application lifecycle management through Managed Service for Apache Flink console and APIs
- KPU-based automatic scaling and resource management
- Service-level configuration management (checkpoints, savepoints, parallelism)
- Integrated CloudWatch monitoring and logging
- IAM-based security and access control

**Data Plane**:

- Flink JobManager and TaskManager processes managed by Managed Service for Apache Flink
- Managed Service for Apache Flink-controlled checkpointing and savepoint storage in managed S3 buckets
- Optimized network and storage configuration for AWS environment
- Automatic fault tolerance and recovery with service-level checkpoint management

### Key Components

1. **Flink Applications**: Your stream processing logic packaged as JAR files, developed using Kiro IDE
2. **KPU Configuration**: Kinesis Processing Units providing standardized resource allocation (1 vCPU, 4GB memory per KPU)
3. **Service-Level Configuration**: Managed Service for Apache Flink-managed settings for checkpoints, savepoints, parallelism, and infrastructure
4. **Application Configuration**: User-controlled runtime parameters, business logic settings, and connector configurations
5. **CloudWatch Integration**: Automatic metrics collection, logs aggregation, and monitoring dashboards

### KPU-Based Resource Model

**Kinesis Processing Units (KPUs)**:

- **You do not pick instance types or manage TaskManagers directly.** MSF abstracts both away — KPU is the only resource unit you configure. Selecting EC2 instance types is not a setting you can change, including via the console. Custom CPU/memory ratios per KPU also are not configurable in MSF.
- Each KPU provides exactly 1 vCPU and 4 GB of memory (standardized resource allocation), plus 50 GB of running application storage.
- Managed Service for Apache Flink automatically scales KPUs based on application throughput and backpressure metrics
- You configure `Parallelism` (total task slots) and `ParallelismPerKPU` (slots per KPU) at the service level; MSF derives `Allocated KPUs = Parallelism / ParallelismPerKPU`
- KPU allocation determines the maximum parallelism and resource capacity available to your application

**Resource Scaling**:

- Automatic horizontal scaling based on real-time throughput and backpressure analysis
- Service-level parallelism configuration through Managed Service for Apache Flink console overrides application defaults
- Managed Service for Apache Flink manages TaskManager allocation and distribution across KPUs automatically
- Pay-per-use pricing model - only pay for the KPUs your application actively uses
- Scaling decisions are made by Managed Service for Apache Flink based on performance metrics, not manual configuration

## Managed Service for Apache Flink Capabilities

### Core Streaming Capabilities

**Stream Processing APIs**:

- DataStream API for low-level stream processing
- Table API for relational stream processing with improved performance
- Flink SQL for declarative stream analytics with expanded function library
- Complex Event Processing (CEP) for pattern detection (note: Flink 2.x requires explicit TypeInformation for CEP pattern output)

**State Management**:

- Managed keyed state with Managed Service for Apache Flink-controlled automatic checkpointing
- Broadcast state for configuration distribution
- RocksDB state backend optimized for Managed Service for Apache Flink environment
- Automatic state cleanup and TTL management
- Service-level checkpoint configuration (interval, retention, storage)

**Time Processing**:

- Event time processing with watermarks (default and only time characteristic in Flink 2.x)
- Processing time for low-latency scenarios
- Custom timestamp extractors and watermark strategies

### AWS Service Integration

**Data Sources**:

- Amazon Kinesis Data Streams
- Amazon MSK (Managed Streaming for Apache Kafka)
- Amazon S3 for batch processing
- Amazon DynamoDB Streams

**Data Sinks**:

- Apache Iceberg
- Amazon S3 with various formats (Parquet, JSON, CSV)
- Amazon DynamoDB for real-time updates
- Amazon OpenSearch Service
- Amazon Data Firehose
- Custom sinks via AWS SDK

For Managed Service for Apache Flink vs self-managed Flink differences, MSF-specific constraints, and common MSF patterns, see [msf-constraints-and-patterns.md](msf-constraints-and-patterns.md).

## Next Steps

After understanding Managed Service for Apache Flink architecture and capabilities:

1. **Environment Setup**: Configure your Kiro IDE development environment for Managed Service for Apache Flink development with Docker containerization
2. **Development Patterns**: Learn Managed Service for Apache Flink-optimized application patterns and templates
3. **Local Development**: Set up Docker-based local testing workflows in Kiro before Managed Service for Apache Flink deployment
4. **Deployment**: Understand Managed Service for Apache Flink deployment procedures and service-level configuration best practices

For detailed guidance on each of these areas, refer to the corresponding guides in the steering directory. All development workflows are optimized for Kiro IDE with Docker-based local development targeting Managed Service for Apache Flink deployment.

# MSF Pricing Calculator

## Overview

Estimate MSF application monthly cost for sizing decisions, budgeting, and optimization analysis. All prices below are us-east-1 — verify against [MSF pricing page](https://aws.amazon.com/managed-service-apache-flink/pricing/) for other regions.

## Pricing Components

| Component | Price (us-east-1) | Notes |
|-----------|-------------------|-------|
| KPU-hour | $0.11 / KPU-hour | Per allocated KPU per running hour |
| Orchestration KPU | $0.11 / hour | 1 additional KPU per running app, **always billed** |
| Running application storage | $0.10 / GB-month | **50 GB per KPU is included free.** Only the bytes *above* `KPU_count × 50 GB` are billable. Do not bill the 50 GB allocation itself. |
| Durable application backups (snapshots) | $0.023 / GB-month | Billed for total snapshot footprint |

There is **no separate charge for application code in S3** (your S3 bucket cost only) and **no per-API-call charge** for Flink REST API or `kinesisanalyticsv2` operations.

## Formula

```
KPU_count   = Parallelism / ParallelismPerKPU
Total_KPUs  = KPU_count + 1                                    # orchestration

KPU_cost     = Total_KPUs × $0.11 × hours_per_month
storage_cost = max(0, state_GB - KPU_count × 50) × $0.10
backup_cost  = total_snapshot_GB × $0.023

monthly_total = KPU_cost + storage_cost + backup_cost
```

24/7 = 730 hours/month. Most production apps stay under the 50 GB/KPU storage included tier.

## Sizing → Cost Reference (24/7, $0 storage)

| Input Rate | Parallelism | PPK | KPUs (+orch) | Monthly |
|-----------|-------------|-----|--------------|---------|
| < 5 MB/s | 2 | 1 | 2+1=3 | $241 |
| 5–20 MB/s | 4 | 1 | 4+1=5 | $402 |
| 20–50 MB/s | 8 | 2 | 4+1=5 | $402 |
| 50–100 MB/s | 16 | 2 | 8+1=9 | $723 |
| 100–200 MB/s | 32 | 2 | 16+1=17 | $1,365 |
| 200–500 MB/s | 64 | 4 | 16+1=17 | $1,365 |
| > 500 MB/s | 128+ | 4 | 32+1=33+ | $2,650+ |

These assume stateless-to-moderate stateful workloads. Stateful workloads (keyed windows with large state) typically require PPK=1, doubling KPU count and cost vs. the stateless equivalent.

## Worked Examples

**Small / development (Parallelism=2, PPK=1, 24/7, negligible state, no snapshots):**

- KPU count: 2 / 1 = 2 application KPUs, plus 1 orchestration KPU = **3 billed KPUs**
- KPU cost: 3 × $0.11 × 730 = **$240.90**
- Running application storage: state ≤ 2 × 50 GB included → **$0** (the 50 GB/KPU allocation is *included*, not separately billable; do **not** multiply 50 × $0.10 per KPU)
- Snapshot cost: $0 (none retained)
- **Total: ~$241/month**

The orchestration KPU is the most common sizing mistake at this scale: it raises the bill from $158 (2 KPUs) to $241 (3 KPUs), a 52% increase. Always include it.

**Medium production (8 parallel, PPK=1, 24/7, 5 GB state):**

- KPUs: 8 + 1 = 9 → 9 × $0.11 × 730 = **$722.70**
- Storage: 5 GB ≤ 8 × 50 GB included → $0
- Snapshots: 5 GB × $0.023 = $0.12
- **Total: ~$723/month**

**Enterprise with autoscaling (avg 16 KPU, peak 32 KPU 20% of time, 60 GB state):**

- Peak: 33 × $0.11 × 146 = $530
- Off-peak: 17 × $0.11 × 584 = $1,092
- Snapshots: 60 GB × $0.023 = $1.38
- **Total: ~$1,623/month**

## Cost Optimization Levers

Ranked by typical impact:

1. **Right-size PPK for workload**: Stateless apps at PPK=2 cost half of PPK=1 at the same parallelism. Rule of thumb: stateless → PPK=2; stateful → PPK=1.
2. **Stop non-prod when idle**: Dev/test apps run 200h/month vs 730h is a 73% reduction. Snapshot before stop, restore on start.
3. **Enable autoscaling for variable load**: 30–50% average reduction for spiky workloads with predictable off-peaks.
4. **Match Parallelism to source partitions**: Idle subtasks waste KPUs. Parallelism > shard/partition count creates them.
5. **Snapshot retention strategy**: Without retention, a streaming app's snapshot footprint grows monotonically.
6. **Always remember the orchestration KPU**: At 1–2 KPU sizing it's 33–50% of the bill — easy to under-estimate.

## Common Mistakes

| Mistake | Impact |
|---------|--------|
| Forgetting orchestration KPU | Under-estimates by 1 KPU = $80/month at 24/7 |
| Confusing Parallelism with KPU count | Wrong with PPK > 1; KPUs = Parallelism / PPK |
| Using us-east-1 prices for other regions | Prices vary; check the pricing page |
| Estimating peak as average | Autoscaling jobs have asymmetric peak/off-peak — calculate separately |
| Ignoring CloudWatch metric cardinality cost | OPERATOR/PARALLELISM metric levels add custom-metric cost (not MSF cost). See [monitoring-and-metrics.md](monitoring-and-metrics.md) |

## References

- [MSF Pricing](https://aws.amazon.com/managed-service-apache-flink/pricing/)
- [resource-optimization.md](resource-optimization.md) — KPU sizing inputs for the calculator
- [scaling-decisions.md](scaling-decisions.md) — cost impact of scaling actions

# Resource Optimization Guide

## Overview

This guide covers KPU sizing, operator parallelism tuning, Amazon Managed Service for Apache Flink configuration overrides, and checkpoint resource impact. Use it when right-sizing a new Managed Service for Apache Flink application or optimizing an existing one based on CloudWatch metrics.

## KPU Sizing Methodology

### CRITICAL: Understanding the KPU Resource Model

Each Kinesis Processing Unit (KPU) provides exactly:

- 1 vCPU
- 4 GB memory (3 GiB JVM heap + 1 GiB reserved for native code allocations including RocksDB, network buffers, and framework overhead)
- 50 GB running application storage

Managed Service for Apache Flink allocates KPUs based on two configuration parameters you set at the service level (console or API):

- **Parallelism**: The total number of parallel task slots for your application (default: 1, default max: 256)
- **ParallelismPerKPU**: The number of task slots per KPU (default: 1, max: 8)

MSF derives the KPU count from these settings:

**Allocated KPUs = Parallelism / ParallelismPerKPU**.

| Parallelism | ParallelismPerKPU | Allocated KPUs | Resources |
|-------------|-------------------|----------------|-----------|
| 4           | 1                 | 4              | 4 vCPU, 16 GB |
| 8           | 1                 | 8              | 8 vCPU, 32 GB |
| 16          | 2                 | 8              | 8 vCPU, 32 GB |
| 16          | 1                 | 16             | 16 vCPU, 64 GB |
| 32          | 2                 | 16             | 16 vCPU, 64 GB |
| 32          | 1                 | 32             | 32 vCPU, 128 GB |

Use ParallelismPerKPU = 1 for most workloads. Increase only when the application is CPU-light and memory-light per task slot (e.g., simple filtering or routing jobs), or for applications with blocking operations (e.g., I/O) where higher values lead to fuller utilization of KPU resources. Higher values pack more task slots per KPU, reducing memory/CPU/storage available per slot.

### Estimating Needed KPU Count

Start with the highest of these three estimates, then add headroom:

**1. CRITICAL: Throughput-based estimate:**

```
base_kpus = (input_record_rate × avg_record_size_bytes × processing_amplification) / throughput_per_kpu
```

- `processing_amplification`: ratio of total bytes processed (including intermediate shuffles) to input bytes. Typically 2–4× for jobs with `keyBy` and windowing.
- `throughput_per_kpu`: start with 5–10 MB/s per KPU for typical ETL workloads. CPU-intensive transformations (regex, JSON parsing, ML inference) reduce this to 1–3 MB/s.

**2. CRITICAL: State-size-based estimate:**

```
base_kpus = total_state_size_gb / usable_memory_per_kpu_gb
```

- `usable_memory_per_kpu_gb`: approximately 2–2.5 GB per KPU after JVM overhead and network buffers (out of 3 GiB heap). The 1 GiB native memory is used by RocksDB and framework overhead. With ParallelismPerKPU = 2, usable memory per slot drops to ~1–1.2 GB.

**3. CRITICAL: Source-parallelism-based estimate:**

```
base_kpus = max(kinesis_shard_count, kafka_partition_count)
```

Source parallelism should match the partition/shard count. If the source has 16 shards, you generally need at least 16 task slots.  

**Final KPU count with headroom:**

```
recommended_kpus = max(throughput_estimate, state_estimate, source_estimate) × 1.3
```

The 1.3× multiplier provides ~30% headroom for checkpoint overhead, traffic spikes, and GC pauses. Round up to the nearest even number for balanced TaskManager allocation.

### Auto-Scaling Behavior

Managed Service for Apache Flink's built-in auto-scaling uses fixed rules based on `containerCPUUtilization`:

- **Scale up**: When `containerCPUUtilization` exceeds 75% for 15 consecutive 1-minute datapoints, Managed Service for Apache Flink doubles `CurrentParallelism` (which increases allocated KPUs).
- **Scale down**: When `containerCPUUtilization` stays below 10% for 360 consecutive 1-minute datapoints (6 hours), Managed Service for Apache Flink halves `CurrentParallelism`. Will never reduce below the configured `Parallelism` setting.
- **During scaling**: Application enters `AUTOSCALING` status with downtime. Only `StopApplication` with `Force=true` is valid.
- Auto-scaling reacts only to CPU, not `heapMemoryUtilization` or backpressure. Scale manually for other metrics.
- Default KPU limit: 64 per application. Request increase via Service Quotas.

**For finer-grained control**, disable built-in auto-scaling and implement custom scaling using CloudWatch alarms and the `UpdateApplication` API.

### Using CloudWatch Metrics to Determine Scaling Direction

After deployment, use these CloudWatch metrics to validate sizing and adjust:

| Metric | Scale Up Signal | Scale Down Signal |
|--------|----------------|-------------------|
| `containerCPUUtilization` | Sustained > 75% over 15 min (triggers auto-scale-up) | Sustained < 10% over 6 hours (triggers auto-scale-down) |
| `heapMemoryUtilization` | Sustained > 80% | Sustained < 40% |
| `backPressuredTimeMsPerSecond` | > 100 ms/s sustained | Consistently 0 |
| `lastCheckpointDuration` | Increasing trend, approaching interval | Stable and well below interval |
| `millisBehindLatest` (Kinesis) | Increasing over time | Stable near 0 |

**Scaling decision process:**

**`heapMemoryUtilization` graduated thresholds**:

- **Healthy:** ≤ 75% — no action needed
- **Scale-up / investigation:** > 80% sustained — investigate state size, TTL, and consider adding KPUs (see [monitoring-and-metrics.md](monitoring-and-metrics.md))
- **Critical alarm:** > 90% — immediate action required; risk of OOM (see [checkpoint-tuning.md](checkpoint-tuning.md) for OOM diagnostic steps)

1. High `containerCPUUtilization` + low `heapMemoryUtilization` → add KPUs (CPU-bound)
2. High `heapMemoryUtilization` + low CPU → increase KPUs or request memory override (memory-bound)
3. High `backPressuredTimeMsPerSecond` → identify bottleneck operator, then scale or optimize
4. Growing `millisBehindLatest` → add KPUs or optimize processing logic

## Operator Parallelism Tuning

### When to Override Default Parallelism

Managed Service for Apache Flink sets application-level parallelism through the service console (`Parallelism` and `ParallelismPerKPU` parameters). In most cases, all operators inherit the configured `Parallelism` as their default. Override per-operator parallelism with `setParallelism()` only when:

- A source operator needs parallelism matched to its partition/shard count
- A CPU-intensive operator needs higher parallelism than the rest of the pipeline
- A lightweight operator (simple filter, static enrichment) can run at lower parallelism to free task slots

**Do not** set application-level parallelism in code for Managed Service for Apache Flink deployments. Managed Service for Apache Flink service-level settings take precedence.

### CRITICAL: Source Parallelism Recommendations

Set source parallelism equal to the number of Kinesis shards or Kafka partitions:

```java
// Kinesis: match shard count
DataStream<Event> events = env
    .fromSource(kinesisSource, watermarkStrategy, "kinesis-source")
    .setParallelism(16)  // Match shard count
    .uid("kinesis-source-uid");
```

If parallelism < partition/shard count, some subtasks handle multiple partitions (uneven load). If parallelism > count, excess subtasks sit idle and waste task slots.

### Per-Operator Parallelism and KPU Interaction

When you set per-operator parallelism, Managed Service for Apache Flink still allocates task slots based on the maximum parallelism across all operators. Operators with lower parallelism use fewer slots; operators with higher parallelism require enough total slots to accommodate them.

**Example**: Parallelism = 16, ParallelismPerKPU = 1 → 16 KPUs, 16 task slots.

```java
// Source: 8 shards → parallelism 8
DataStream<Event> events = env
    .fromSource(kinesisSource, watermarkStrategy, "kinesis-source")
    .setParallelism(8)
    .uid("kinesis-source-uid");

// CPU-heavy processing: inherits default parallelism of 16
DataStream<Result> results = events
    .keyBy(Event::getKey)
    .process(new HeavyProcessor())
    .uid("heavy-processor-uid");

results.sinkTo(sink).uid("sink-uid");
```

## Managed Service for Apache Flink Configuration Overrides

### Parameters Overridable via AWS Support

Managed Service for Apache Flink manages most infrastructure configuration automatically. However, certain parameters can be overridden by opening an AWS support case. These include:

| Parameter | Default | Override Range | Use Case |
|-----------|---------|---------------|----------|
| JVM heap size | 3 GiB (~75% of KPU memory) | Custom | Applications with large in-memory caches or high object churn |
| TaskManager native memory | 1 GiB (~25% of KPU memory) | Custom | Adjusting RocksDB vs heap balance |
| RocksDB block cache size | Auto-configured | Custom size | Large state with frequent random reads |
| RocksDB write buffer count | Default | 2–6 | High write-throughput state workloads |
| Network buffer memory | Auto-configured | Custom size | Jobs with high fan-out or many network channels |
| State backend type (RocksDB vs. HashMap) | RocksDB | | Jobs with lightweight state that can stay in-memory and benefit from faster in-memory performance |

### Process for Requesting Overrides

1. Gather diagnostic evidence: CloudWatch metrics showing the resource constraint (heap utilization, GC time, checkpoint duration trends)
2. Open an AWS support case under "Managed Service for Apache Flink"
3. Include: application ARN, current KPU count, the specific parameter to override, the requested value, and the diagnostic evidence
4. AWS support applies the override at the service level — no application code changes needed
5. After the override is applied, Managed Service for Apache Flink restarts the application to pick up the new configuration

For checkpoint impact on resources (checkpoint size and memory, frequency vs CPU/network, duration exceeding interval, OOM/GC diagnostic steps), see [checkpoint-tuning.md](checkpoint-tuning.md).

# Scaling Decision Framework

## Overview

Decision matrix for choosing the right scaling action based on observed bottleneck. For initial KPU sizing methodology and CloudWatch trend signals, see [resource-optimization.md](resource-optimization.md). This guide is for in-flight scaling decisions on running applications.

## Decision Matrix

```
What is the bottleneck?
│
├─► CPU > 70% sustained or backpressure > 500ms/s
│   → INCREASE Parallelism (adds KPUs, restart required)
│
├─► OOM errors or heapMemoryUtilization > 85%
│   → DECREASE ParallelismPerKPU (more memory per subtask)
│   → Same total parallelism, more KPUs, lower throughput-per-dollar
│
├─► Variable/spiky load + can tolerate restart-on-scale
│   → ENABLE AutoScalingEnabled=true
│   → MSF reacts on CPU only; 5-15 min reaction; restart per event
│
└─► CPU < 30% sustained, idle > 500ms/s, no backpressure
    → DECREASE Parallelism (over-provisioned)
    → Validate with 60+ minute trend before reducing
```

## Scaling Impact Table

| Change | KPU Effect | Memory Per Subtask | Cost Impact |
|--------|-----------|---------------------|-------------|
| Parallelism 4 → 8, PPK=1 | 4 → 8 KPU | 4 GB (unchanged) | 2× |
| Parallelism 4 → 8, PPK=2 | 2 → 4 KPU | 2 GB (unchanged) | 2× |
| PPK 2 → 1, Parallelism=8 | 4 → 8 KPU | 2 → 4 GB | 2× |
| PPK 1 → 2, Parallelism=8 | 8 → 4 KPU | 4 → 2 GB | 0.5× |
| Parallelism 8 → 4, PPK=1 | 8 → 4 KPU | 4 GB (unchanged) | 0.5× |

KPU formula: `KPU = Parallelism / ParallelismPerKPU`. Add 1 orchestration KPU for total billed.

## Pre-Scaling Validation

Always validate the bottleneck against trends, not point-in-time values, before scaling. Pull the last 6 hours of:

- `containerCPUUtilization` — Maximum (hottest container)
- `backPressuredTimeMsPerSecond` — Average and Maximum
- `idleTimeMsPerSecond` — Average and Minimum
- `heapMemoryUtilization` — Maximum
- `lastCheckpointDuration` — trend (rising = pressure)

If signals conflict (high CPU but also high idle, or backpressure with low CPU), the bottleneck is downstream — investigate the operator graph via Flink Dashboard before scaling. Adding KPUs to a sink-bound or skew-bound job wastes money. See [first-fault-isolation.md](first-fault-isolation.md) for live diagnosis via the Dashboard.

## AutoScalingEnabled Behavior

MSF built-in autoscaling is CPU-only with fixed thresholds:

- **Scale up:** `containerCPUUtilization > 75%` for 15 consecutive 1-min datapoints → doubles `CurrentParallelism`
- **Scale down:** `containerCPUUtilization < 10%` for 360 consecutive 1-min datapoints (6h) → halves `CurrentParallelism`, never below configured `Parallelism`
- **Each event triggers a full restart** (10–30s downtime). Backpressure, lag, and memory pressure do **not** trigger autoscaling.
- During scaling: status is `AUTOSCALING`. Only `stop-application --force` is valid.

For backpressure-driven, lag-driven, or memory-driven autoscaling: disable `AutoScalingEnabled` and implement custom scaling via CloudWatch alarms → Lambda → `update-application`.

## Stateful vs Stateless ParallelismPerKPU

| Workload | ParallelismPerKPU | Why |
|----------|-------------------|-----|
| Stateful (keyed windows, joins, large RocksDB state) | 1 | Each subtask needs full 4 GB; sharing causes RocksDB contention and OOM |
| Stateless transforms (map, filter, simple routing) | 2 | Half memory per subtask is fine; doubles compute density per KPU |
| I/O-blocking (async lookups, slow sinks) | 2–4 | Subtasks spend most time blocked; pack more per KPU to fill CPU |
| Source operators matched to shards/partitions | Match to source count | Use `setParallelism()` on the source only |

## Scaling Guardrails

- ❌ Cannot scale during a transitional state (STARTING, UPDATING, STOPPING, AUTOSCALING)
- ❌ Cannot scale during another in-flight update — sequence operations
- ⚠️ Every scaling change triggers a restart with 10–30s downtime (varies with state size and snapshot recency)
- ⚠️ Take a snapshot before scaling for rollback safety
- ⚠️ Scaling beyond 64 KPU requires a Service Quotas increase (default limit)
- ⚠️ Setting Parallelism > source partition/shard count creates idle subtasks — wasted KPUs

## Anti-Patterns

| Anti-Pattern | Why It Fails |
|-------------|--------------|
| Scaling up to fix checkpoint failures | Checkpoint failures are usually serialization, S3 perms, or alignment timeout — not capacity |
| Scaling up to fix data skew | Hot keys still go to the same subtask. Fix with `rebalance()` or pre-splitting keys |
| Scaling up to fix slow sink | Sink is the bottleneck regardless of KPU count. Optimize sink (batching, async I/O) or switch sink |
| Scaling during active restart loop | Restart loops persist until root cause is fixed; new KPUs join the loop |
| Reducing PPK and Parallelism in separate updates | Each triggers a restart; two sequential updates = double downtime. Combine in a single `update-application` call |
| Scaling on a single high CPU spike | Spikes from checkpointing or GC are normal; require sustained 30+ minute trend |

## References

- [MSF Auto-Scaling Behavior](https://docs.aws.amazon.com/managed-flink/latest/java/how-scaling.html#how-scaling-auto)
- [resource-optimization.md](resource-optimization.md) — KPU sizing methodology and trend-based signals
- [first-fault-isolation.md](first-fault-isolation.md) — diagnose before scaling

# Serialization Best Practices

## Overview

This guide covers serialization best practices for Managed Service for Apache Flink applications, including the performance hierarchy of serializer types, POJO and Tuple usage, Avro and Protobuf integration, Kryo avoidance, state serialization considerations, and common anti-patterns.

For general development patterns and application structure, see [best-practices.md](best-practices.md). For state management guidance, see [state-management.md](state-management.md).

Code examples in this guide use Flink 2.2 APIs by default, which are also compatible with Flink 1.20 unless noted otherwise. See `flink-2x-migration.md` for the complete migration reference.

## Performance Hierarchy: Choose the Right Serializer

Flink serialization performance (fastest to slowest):

1. **Flink Tuples/Rows** - Fastest (direct field access, no reflection)
2. **POJOs** - Fast (~30% slower than tuples, supports schema evolution)
3. **Protobuf** - Good performance (~30% slower than POJOs)
4. **Avro Specific** - Moderate (~50% slower than POJOs)
5. **Avro Generic/Thrift** - Slower (~70% slower than POJOs)
6. **Kryo** - Avoid (50%+ performance penalty). Kryo registration convenience methods are removed from `StreamExecutionEnvironment` in Flink 2.x; avoid Kryo entirely.

## POJO Serialization for Managed Service for Apache Flink

```java
// Recommended: Flink POJO for optimal performance with schema evolution
public class OptimizedEvent {
    // All fields must be public or have public getters/setters
    public String eventId;
    public long timestamp;
    public String userId;
    public EventType type;
    
    // Required: public no-argument constructor
    public OptimizedEvent() {}
    
    public OptimizedEvent(String eventId, long timestamp, String userId, EventType type) {
        this.eventId = eventId;
        this.timestamp = timestamp;
        this.userId = userId;
        this.type = type;
    }
}

// Enum types work well with POJO serialization
public enum EventType {
    USER_ACTION, SYSTEM_EVENT, ERROR_EVENT
}
```

## Tuple Types for Maximum Performance

```java
// Use when performance is critical and schema evolution is not needed
DataStream<Tuple4<String, Long, String, Integer>> events = source
    .map(event -> Tuple4.of(event.getId(), event.getTimestamp(), 
                           event.getUserId(), event.getCount()));

// Access fields by position (f0, f1, f2, f3)
events.keyBy(tuple -> tuple.f2) // Key by userId (f2)
      .process(new TupleProcessor());
```

## Avro for External Integration

```java
// Use Avro when integrating with external systems or when advanced schema evolution is needed
public class AvroEventProcessor extends ProcessFunction<SpecificRecordBase, ProcessedEvent> {
    
    @Override
    public void processElement(SpecificRecordBase avroEvent, Context ctx, 
                              Collector<ProcessedEvent> out) {
        if (avroEvent instanceof UserEvent) {
            UserEvent userEvent = (UserEvent) avroEvent;
            ProcessedEvent result = new ProcessedEvent();
            result.setUserId(userEvent.getUserId().toString());
            result.setTimestamp(userEvent.getTimestamp());
            out.collect(result);
        }
    }
}

// Configure Avro serialization
// Note: enableForceAvro() is available in Flink 1.20 but removed in 2.x.
// For Flink 2.2, use AvroTypeInfo explicitly in state descriptors instead.
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// Flink 1.20 only:
// env.getConfig().enableForceAvro();
```

## Protobuf Integration

The recommended approach for Protobuf is to convert Protobuf messages to Flink POJOs at the ingestion boundary. This avoids Kryo entirely, giving you fast serialization, schema evolution support, and state compatibility across Flink major versions.

```java
// Recommended: Convert Protobuf to POJO at the boundary, avoiding Kryo entirely
DataStream<MyEvent> events = protobufSource
    .map(proto -> new MyEvent(
        proto.getEventId(),
        proto.getTimestamp(),
        proto.getUserId()));
// MyEvent is a Flink POJO (public fields + no-arg constructor) — fast serialization, schema evolution, no Kryo
```

> **Legacy note — not recommended for new applications:** If you must use Protobuf objects directly in state, you can register them with Kryo via `env.getConfig()`. However, Kryo has a 50%+ performance penalty and Kryo-serialized state does not migrate from Flink 1.x to 2.x. Convenience registration methods on `StreamExecutionEnvironment` are removed in Flink 2.x.
>
> ```java
> // Not recommended — use POJO conversion instead:
> env.getConfig().registerTypeWithKryoSerializer(
>     MyProtobufMessage.class,
>     ProtobufSerializer.class
> );
> ```

## Avoiding Kryo Fallbacks

### Why Kryo Fallback Matters on MSF

If Flink can't recognize a type as a POJO/Tuple/Avro/Protobuf, it silently falls back to Kryo. On MSF this has three consequences worth treating as blockers, not warnings:

- **~50% performance penalty** vs. POJO serialization, plus larger serialized objects on the wire and in state. On a high-throughput keyed pipeline this dominates per-record cost.
- **Larger checkpoint and shuffle bytes.** Inflated checkpoint size lengthens the checkpoint window and pushes more data across cross-AZ network paths inside MSF.
- **Kryo-serialized state does not migrate from Flink 1.x to 2.x.** This is a hard blocker for in-place version upgrades — see [flink-2x-migration.md](flink-2x-migration.md) for the migration path. Plan to eliminate Kryo *before* the 1→2 upgrade, not after.

### Fail Fast in Development

```java
// Monitor for Kryo fallbacks in logs - these indicate performance issues
// Log message: "Class ... cannot be used as a POJO type because not all fields are valid POJO fields"

// To detect Kryo usage, disable it temporarily during development
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.getConfig().disableGenericTypes(); // Throws exception if Kryo would be used
// This will fail with: "Generic types have been disabled in the ExecutionConfig"
```

Run with `disableGenericTypes()` enabled locally as part of every PR build so Kryo fallbacks fail the build, not production.

## Last Resort: Kryo Type Registration

> **Warning:** Prefer converting to Flink POJOs or Tuples instead of registering Kryo serializers. Kryo-serialized state does not migrate across Flink major versions. Convenience registration methods on `StreamExecutionEnvironment` are removed in Flink 2.x — use `env.getConfig()` methods instead. Use this only when migrating away from Kryo is not yet feasible.

```java
// Last resort — register frequently used types to avoid class name serialization overhead IF you use Kryo
// In Flink 2.x, use env.getConfig() methods (env-level convenience methods are removed)
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

env.getConfig().registerKryoType(CustomEvent.class);
env.getConfig().registerKryoType(ProcessingResult.class);

env.getConfig().registerTypeWithKryoSerializer(
    ComplexObject.class, 
    CustomKryoSerializer.class
);
```

## State Serialization Considerations

```java
// For state objects, prioritize schema evolution support
public class StatefulProcessor extends KeyedProcessFunction<String, Event, Result> {
    
    // Use POJO or Avro for state that needs to evolve
    private transient ValueState<EventAggregate> aggregateState; // POJO - good performance + evolution
    
    // Use primitive types for simple state
    private transient ValueState<Long> counterState; // Primitive - fastest
    
    @Override
    public void open(OpenContext openContext) throws Exception {
        // POJO state descriptor - supports schema evolution
        ValueStateDescriptor<EventAggregate> aggregateDescriptor = 
            new ValueStateDescriptor<>("aggregate", EventAggregate.class);
        aggregateState = getRuntimeContext().getState(aggregateDescriptor);
        
        // Primitive state descriptor - fastest serialization
        ValueStateDescriptor<Long> counterDescriptor = 
            new ValueStateDescriptor<>("counter", Long.class);
        counterState = getRuntimeContext().getState(counterDescriptor);
    }
}
```

## Anti-Patterns: Serialization Performance Killers

```java
// AVOID: Default Java Serialization (extremely slow)
public class SlowEvent implements Serializable {
    // Java serialization is 10x+ slower than POJO serialization
}

// AVOID: Complex nested objects without proper POJO structure
public class BadEvent {
    private Map<String, Object> data; // Generic Object causes Kryo fallback
    private List<SomeInterface> items; // Interface types cause Kryo fallback
}

// AVOID: Missing no-argument constructor (causes Kryo fallback)
public class InvalidPOJO {
    public String field;
    
    // Missing no-arg constructor - will use Kryo instead of POJO serializer
    public InvalidPOJO(String field) {
        this.field = field;
    }
}

// AVOID: Private fields without getters/setters (causes Kryo fallback)
public class AlmostPOJO {
    private String secretField; // No getter/setter - not a valid POJO field
    public String publicField;   // This is fine
    
    public AlmostPOJO() {} // Constructor is correct
}
```

# State Management Best Practices

## Overview

This guide covers state management best practices for Managed Service for Apache Flink applications, including efficient state usage, TTL configuration, state type selection, and Managed Service for Apache Flink-specific state management considerations.

For general development patterns and application structure, see [best-practices.md](best-practices.md). For serialization guidance, see [serialization-guide.md](serialization-guide.md).

Code examples in this guide use Flink 2.2 APIs by default, which are also compatible with Flink 1.20 unless noted otherwise. See `flink-2x-migration.md` for the complete migration reference.

## Efficient State Usage with Managed Service for Apache Flink

- Estimate state size ahead of time to ensure state will remain bounded over time.
- Enable state TTL to ensure state gets cleaned up automatically if not cleaned up manually.
- Use the correct state type for each use case, and perform updates to state in performant way (e.g. make updates to a map key, rather than replacing the entire map).
- AVOID: Storing large objects or unbounded collections in state

### Pick the Right State Type — `MapState` vs `ValueState<Map>`

Use `MapState<K, V>` whenever you need per-key updates inside a logical map. Storing a `Map<K, V>` inside `ValueState<Map<K, V>>` and reading-mutating-writing it on every event is `O(map size)` per access — RocksDB has to deserialize every entry, your code mutates one, and the whole map gets re-serialized and written back. `MapState` is `O(1)` per `put`/`get`/`remove`: each map entry maps to its own RocksDB key, so only the touched entry is read or written.

There is also a state-migration consequence specific to MSF: nested generic collections inside `ValueState` (e.g. `ValueState<Map<String, MyType>>`) typically fall back to Kryo serialization, and **Kryo-serialized state does not migrate from Flink 1.x to 2.x.** `MapState` uses a dedicated `MapSerializer` that does carry across the upgrade. See [serialization-guide.md](serialization-guide.md) for the wider Kryo guidance and [flink-2x-migration.md](flink-2x-migration.md) for the migration impact.

```java
public class OptimizedKeyedProcessor extends KeyedProcessFunction<String, Event, Result> {
    
    // Use appropriate state types
    private transient ValueState<EventAggregate> aggregateState;
    
    @Override
    public void open(OpenContext openContext) throws Exception {
        ValueStateDescriptor<EventAggregate> aggregateDescriptor = 
            new ValueStateDescriptor<>("aggregate", EventAggregate.class);
        
        // TTL configuration - application-level concern
        aggregateDescriptor.enableTimeToLive(StateTtlConfig.newBuilder(Duration.ofHours(24))
            .setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite)
            .setStateVisibility(StateTtlConfig.StateVisibility.NeverReturnExpired)
            .build());
        
        aggregateState = getRuntimeContext().getState(aggregateDescriptor);
    }
    
    @Override
    public void processElement(Event event, Context ctx, Collector<Result> out) throws Exception {
        // Efficient state access patterns
        EventAggregate current = aggregateState.value();
        if (current == null) {
            current = new EventAggregate();
        }
        
        // Update state efficiently
        current.update(event);
        aggregateState.update(current);
        
        // Use timers for handling events that occur after the input event - e.g. in this case we want to trigger the output an hour after the input occurs
        ctx.timerService().registerEventTimeTimer(event.getTimestamp() + 3600000); // 1 hour
    }

    @Override
    public void onTimer(long timestamp, OnTimerContext ctx, Collector<Result> out) throws Exception {
        // Handle timer firing - cleanup or emit final results
        EventAggregate current = aggregateState.value();
        if (current != null) {
            // Emit final result or perform cleanup
            out.collect(new Result(ctx.getCurrentKey(), current.getFinalValue()));
            // Clear state after processing to re-initialize if needed
            aggregateState.clear();
        }
    }
}
```

## Managed Service for Apache Flink State Management

Managed Service for Apache Flink service handles:

- State backend configuration (RocksDB with S3 for checkpoints/savepoints)
- Checkpoint storage and retention
- Savepoint management through console
- State size monitoring and alerting
Application code should NOT configure these aspects