Your Cloud Data Platform POC Has an Expiry Date
Hi Folks, I had shared a post on LinkedIn comparing a few Snowflake and Databricks capabilities with what was available around mid-2023. Our earlier warehouse POC covered Athena, Dremio, Redshift and Snowflake. Going through the documentation again, there are a few things I would revisit before usin

g those findings for another decision. I will keep this to the same three areas: join elimination, result downloads and Iceberg. Note: The current capabilities below are from public documentation checked on November 13, 2026, not a fresh benchmark. The earlier POC observations apply to the configurations we tested then; we have not tested the newer capabilities and integrations described here.
Let us take a sales report which only needs total revenue. The BI tool may still generate a join between the sales and customer tables because that relationship exists in its model. If the query doesn't use any customer attributes, the database may be able to skip that join. It needs valid key relationships to establish that doing so won't change the result.
Duplicate customer keys, for example, could multiply the sales rows. Snowflake had documented join-elimination improvements by July 2022. Its RELY constraints let the optimizer use declared relationships for qualifying optimizations. Snowflake documentation, 2022 release history.
For the Databricks pattern I discussed, I couldn't find documented support at the mid-2023 point. Primary-key RELY is now documented from Runtime 14.2, released in November 2023. Foreign-key RELY followed in Runtime 15.4 and the September 2024 SQL release. Requirements, Runtime 14.2, SQL release history.
So this is something I would no longer keep as a Snowflake-only advantage. Redshift also documents using primary and foreign keys to remove redundant joins. I haven't established when that was introduced. Redshift documentation.
The supported Databricks compute and table types need checking; classic SQL warehouses are excluded. Also, declaring a key doesn't make the data valid. Incorrect key declarations can lead to incorrect results. For a large BI extract, query execution may finish while the client is still downloading rows.
We looked at this during the earlier evaluation because the compute behavior during that download can affect cost. Snowflake persists completed results separately from the warehouse cache. Its result-storage and suspension behavior allow later retrieval without keeping warehouse compute active solely for that retrieval. Auto-suspend settings and other running work still matter.
Persisted results, warehouse suspension. Databricks has Cloud Fetch to download larger results directly from cloud storage. This was available before mid-2023. Its current monitoring documentation also says SQL warehouses remain active while queries are running or fetching results.
Cloud Fetch, warehouse monitoring. When we tested Redshift during the earlier evaluation, we had to keep the compute running until the client finished downloading the results. We were therefore paying for the full extract duration, not just query execution. We have not repeated that test on current Redshift or Databricks configurations.
The earlier Redshift finding applies to the setup we tested; todayβs behavior and cost need checking with the warehouse, connector and workload we intend to use. Iceberg was one of the formats we were following in 2023. Snowflake Iceberg Tables became generally available in June 2024. Since then, the available integration options have expanded.
Snowflake release. The important detail is where the table's catalog is managed: Scenario What the documentation supports Another engine accessing Snowflake-managed Iceberg Read and write through Horizon Catalog Snowflake accessing supported external Iceberg REST catalogs Read and write through the documented integration Another engine accessing Unity Catalog-managed Iceberg Read and write through the Iceberg REST Catalog API Databricks accessing foreign Iceberg tables managed outside Unity Catalog Read-only through the native foreign-table path Redshift accessing supported Iceberg tables registered in AWS Glue Create, insert, update, delete and merge, with the documented requirements Snowflake-managed access, external-catalog writes, Databricks Iceberg, Redshift Iceberg SQL. Multiple engines do not necessarily need to write to the same table. The distinction matters when a team wants to keep its existing catalog and use another engine for transformations.
Databricksβ native foreign Iceberg integration is documented as read-only; external engines can read and write Unity Catalog-managed Iceberg through its REST API. Whether that restriction matters depends on the intended architecture. Redshift has changed considerably here. Iceberg querying became generally available in November 2023, append-only writes in November 2025, and UPDATE, DELETE and MERGE in April 2026.
That old βnot supportedβ entry needs another look. Querying, inserts, row-level changes. Snowflake also added on-demand table registration for catalog-linked databases on November 3, 2026. A remote table can be registered when referenced, without waiting for the next catalog sync.
Release note. Dremio was part of the original POC. SAP completed its acquisition on July 6, 2026, and has described how Dremio technology fits into SAP Business Data Cloud's Iceberg and open-catalog direction. For teams combining SAP and non-SAP data, I would include that in another evaluation.
Acquisition, SAP update. Microsoft Fabric could be considered as well, particularly by teams already using Power BI. It wasn't in the original POC. Its Warehouse and Lakehouse experiences around OneLake give us another option to evaluate.
Microsoft overview. With AI in the mix, I feel product teams are shipping features faster than ever. As an architect, I still need to look further ahead: how the company works, the people who will maintain the platform, business direction, contracts and the overall financial commitment. I wouldn't change platforms just because a feature was released.
But I also wouldn't use a limitation from an old POC to rule one out without checking it again. If you have recently tested these result-download or Iceberg scenarios on Databricks or Redshift, please share your experience. I would be interested to know what you observed.
Key Takeaways
- β’I will keep this to the same three areas: join elimination, result downloads and Iceberg.
- β’This story was reported by Dev.to, covering developments in the dev space.
- β’AI advancements continue to reshape industries β read the full article on Dev.to for complete coverage.
π Continue reading the full article:
Read Full Article on Dev.to βShare this article



