SQL and Data Blog

Data Governance, gsp, SQLFlow

Why DataHub Loses Column-Level Lineage on dbt Deduplication Macros — and How to Recover It

DataHub silently drops column-level lineage on the dbt-utils deduplicate macro because of how sqlglot's column resolver handles ARRAY_AGG + struct unpack. Here's why — and an open-source post-processor that recovers the missing lineage.

James
May 1, 2026

Data Governance, gsp, SQL language, SQL skills, SQLFlow

BigQuery Column-Level Lineage Without Metadata: Inferring STRUCT and ARRAY Types from SQL Alone

How a BigQuery-aware SQL parser can extract column-level lineage from ARRAY<STRUCT>, SELECT AS STRUCT, and array_agg(row) patterns without any catalog metadata — and where generic parsers silently fail.

James
May 1, 2026

gsp, SQL language, SQLFlow

OpenMetadata + MSSQL Stored Procedures: Why Your Lineage Is Silently Empty — and How to Fix It

OpenMetadata issues #16737, #25299, and #17586 report zero lineage from MSSQL stored procedures. We analyze three failure patterns — BEGIN/END blocks, temp table chains, and square bracket identifiers — with real SQL from the community, and show how gsp-openmetadata-sidecar recovers full column-level lineage.

James
April 21, 2026

gsp, SQLFlow

DataHub #15327: Why Power BI M-Language Queries Produce Zero Lineage — 4-Query Deep Analysis

Deep analysis of 4 Power BI M-language queries from DataHub issue #15327. Both patterns solved: Pattern A (M navigation chains) produces table-level lineage; Pattern B (Value.NativeQuery with embedded SQL) produces column-level lineage with 6 column mappings traced through expressions.

James
April 20, 2026

gsp, sql, SQLFlow

Why Power BI SQL Comments Break Your DataHub Lineage — and How to Fix It

Power BI encodes newlines as #(lf) in M-language SQL. When DataHub parses queries with -- comments, it silently drops all subsequent JOINs from lineage. The gsp-datahub-sidecar recovers every missing relationship.

James
April 19, 2026

gsp, sql, SQLFlow

Why Your DataHub BigQuery Lineage Silently Breaks on Procedural SQL — and How to Fix It

DataHub's BigQuery ingestion silently loses lineage on procedural SQL (DECLARE, IF/THEN, CALL). This post explains why, and shows how to recover the missing lineage using an open-source sidecar tool.

James
April 18, 2026

sql, SQL language, SQLFlow

Analyzing Oracle PL/SQL Dependencies Before Your Snowflake Migration

Oracle-to-Snowflake conversion tools handle syntax translation, but skip dependency analysis. Learn how to use GSP and SQLFlow to map PL/SQL call graphs, column-level lineage, and vendor-specific constructs before you start converting.

James
April 6, 2026

gsp, sql, SQL language

GSP vs JSQLParser vs sqlglot — SQL Parser Comparison 2026

We tested General SQL Parser, JSQLParser, and sqlglot against 14 real-world SQL patterns across 6 dialects. Here are the verified results — including where each parser shines and where it struggles with Oracle, stored procedures, and BigQuery procedural SQL.

James
April 6, 2026

gsp, sql, SQL language

Why Most SQL Lineage Tools Skip Stored Procedures (And What You Can Do About It)

Most SQL lineage tools silently skip stored procedures. Learn why PL/SQL, T-SQL, and PL/pgSQL procedural code breaks typical parsers, and how General SQL Parser extracts column-level lineage through stored procedures -- not just around them.

James
April 5, 2026

Uncategorized

Understanding SQLFlow Job: Automate Your Data Lineage Analysis at Scale

When dealing with large-scale data environments, analyzing data lineage manually becomes impractical. You might have hundreds of SQL files across different repositories, databases, and ETL processes. This is where SQLFlow Job comes into play – a powerful feature that enables…

James
February 5, 2026