Skip to content

2.13.0 integration candidate

Status: unreleased. These notes describe the release/2.13.0 branch in draft PR #111. The published baseline is 2.12.1; pip install seeknal installs a published package, not this candidate. Version metadata on the integration branch is already 2.13.0.

Included behavior

  • Conditional Iceberg upsert: matched rows are accepted only when when_matched_update_if is true. False and NULL preserve the stored row. compare_columns declares additional target columns available to the predicate. New keys insert by default or are excluded with when_not_matched: skip.
  • Configuration parity: conditional options propagate through profiles, YAML targets, stackable Python decorators, the typed pipeline MaterializationConfig, and IcebergStoreOutput. Unset typed values inherit defaults. YAML target dictionaries can explicitly clear an inherited predicate together with compare_columns: [].
  • Backfills and parameters: --date anchors date functions, source paths are templated, and Python decorator parameters reach ctx.params. Functional parameter changes also invalidate feature-group and SOA caches.
  • Lakekeeper configuration: LAKEKEEPER_URI is canonical and LAKEKEEPER_URL remains a fallback alias. If both differ, URI wins.
  • Empty SQL results: a SELECT returning zero rows replaces the previous intermediate parquet with a typed empty file. Downstream nodes receive the current empty result; publication errors fail the node.
  • Session paths: tenant/session identifiers must be valid single path segments whose resolved location is strictly below their storage base.
  • Installed CLI documentation: wheel/sdist packages include CLI topics; installed package topics take precedence over a current project's docs. Read the new conditional upsert topic with seeknal docs iceberg-upsert.

The candidate integrates PRs #95, #107, and #110 plus the previously uncommitted conditional-upsert work. PRs #92, #106, and #109 are excluded: configurable Ask limits/model sampling, gateway-poll resilience changes, and the WAF guard are not additions in this candidate. See the component provenance.

Upgrade considerations

Read Upgrading to the candidate before evaluating the branch against an existing project. In particular:

  • Missing {{ env:VAR }} values without defaults now fail DAG construction for transform SQL that previously reached DuckDB unresolved.
  • Expect a one-time rerun for Python nodes with decorator parameters and feature-group/SOA nodes with parameters. Undeclared Python CLI keys do not automatically invalidate every node.
  • Conditional writes replace complete rows; they do not implement partial updates, SCD Type 2, CDC deletes, or deletion of keys absent from the batch.
  • On conflict, reconcile the concurrent write and start a new invocation so the predicate is evaluated again. For commit_unknown, inspect the target before retrying; automatic generic retries are suppressed for both outcomes.

Compatibility and verification

The package requires Python 3.11+. The existing dependency floor remains DuckDB 1.4.4 and PyArrow 18.1; advanced Iceberg writes check for PyIceberg 0.10.0 or newer at runtime even though the general package dependency allows older PyIceberg. These floors are not a claim that every combination was tested. max_batch_bytes limits the incoming Arrow batch, not all join/writer memory; conditional reads do not guarantee file or partition pruning.

Verification on integration head 7570217:

Check Observed result
Linux CI 5,008 passed, 80 skipped, including the 2,400-row synthetic composite-key workload
Local focused workflow/pipeline/session/docs tests 1,524 passed, 4 skipped
Loopback REST/Parquet acceptance 36 passed, including date-templated empty-output upsert preserving its snapshot
PyIceberg 0.12 matrix 39 passed; large macOS workload excluded from this matrix
Native empty append on a local REST/Parquet table 0 appended rows; existing data and snapshot unchanged
Package and installed docs Wheel/sdist build; 39 bundled CLI Markdown files (38 listed topics plus index.md); three docs commands verified outside the checkout
Multiarch worker build linux/amd64 and linux/arm64 passed; no image publication from the PR workflow

The counts above record that baseline; the subsequent documentation update adds the upgrade topic (40 bundled Markdown files, 39 listed topics). Existing CI excludes Prefect integration and tests/e2e; the skipped count does not imply those paths were tested. All catalog checks above used synthetic local data, without a production catalog or model requests. Earlier implementation receipts are retained as historical evaluation evidence.

Known limits

On macOS ARM, the 2,400-row composite-key case terminates a native process with SIGILL/SIGBUS during the target-key scan with PyArrow 23.0.0, on both tested PyIceberg 0.10 and 0.12 environments. Smaller local cases pass; the exact large case passes on Linux CI. The cause is unresolved: Linux success does not mean the macOS issue is fixed.

The project type check still reports three pre-existing SessionStore.list annotation errors, also present on main. They are separate from the native crash. Core Ruff checks introduced no diagnostics over the corresponding main files. No compatibility or type-check guarantee beyond this evidence is implied.