{"id":1751,"date":"2022-05-07T21:15:23","date_gmt":"2022-05-08T02:15:23","guid":{"rendered":"https:\/\/www.dpriver.com\/blog\/?p=1751"},"modified":"2022-05-22T19:06:18","modified_gmt":"2022-05-23T00:06:18","slug":"gudu-sqlflow-whats-it-why-it-important","status":"publish","type":"post","link":"https:\/\/www.dpriver.com\/blog\/2022\/05\/gudu-sqlflow-whats-it-why-it-important\/","title":{"rendered":"Gudu SQLFlow : What’s it and why it is important?"},"content":{"rendered":"

## What’s Gudu SQLFlow?<\/h2>\n

In large data warehouses and data lakes, complete data lineage<\/a><\/strong> can be used for data traceability, impact analysis of table and column changes, proof of data compliance, data quality checks, and more.\u00a0So what exactly is data lineage?\u00a0<\/span>According to Wikipedia, data lineage includes where the data came from, what happened, and where it moved over time.\u00a0\u00a0Gudu SQLFlow<\/strong> gives complete data lineage by analyzing SQL scripts.\u00a0It can handle SQL statements of more than 20 kinds of databases, including complex stored procedures and dynamic SQL statements.\u00a0With the powerful SQL processing capability of the self-developed general-purpose SQL parser, Gudu SQLFlow is one of the most popular data lineage tools<\/a><\/strong> available on the market today.\u00a0<\/span><\/p>\n

Gudu SQLFlow<\/figcaption><\/figure>\n

Gudu SQLFlow generates data lineages by analyzing SQL queries and stored procedures.\u00a0Entities in the data lineage model include tables, columns, functions, relationships, and other entities. The combination of entities and dataflows shows the lineage from one table\/column to another.<\/p>\n

### 1. Data flow unit<\/h3>\n

A data flow unit includes source entities, target entities, and the type of data flow between them.<\/p>\n

<\/p>\n

This is the data flow generated for the above SQL query.<\/p>\n

<\/p>\n

<\/p>\n

#### 1.1 Source and target entities<\/h4>\n

Source and target entities often reference tables, views, and other relationships, such as common table expressions, result sets generated during query execution. It may also refer to files in HDFS systems etc.<\/p>\n

#### 1.2 Types of data flow<\/h4>\n

There are two types of data flow: direct data flow and indirect data flow.<\/p>\n

##### 1.2.1 Direct data flow<\/h5>\n

Direct data flow means that the data of the target entity comes directly from the source entity. In the diagram above, RS-1.FirstName directly from Person.FirstName.<\/p>\n

Arrows are used to represent direct data flow in the diagram:\u00a0<\/p>\n

##### 1.2.2 Indirect data flow<\/h5>\n

Indirect data flow means that the data of the target column does not come from the source column, but the data of the source column\/table affects the result data of the target column.
\nDashed arrows are used to represent indirect data flow in the diagram:\u00a0<\/p>\n

Source columns in indirect data flow typically appear in the following clauses:<\/p>\n

\n
- Group by clause<\/li>\n
- Where clause<\/li>\n
- Winddows function<\/li>\n
- Join condition<\/li>\n<\/ul>\n

  <\/p>\n

  The values \u200b\u200bof COUNT() and SUM(SAL) are affected by the value of the deptno column in the group by clause. Therefore, the indirect data flow will be created as follows:<\/p>\n

  <\/p>\n

  <\/p>\n

  For other indirect data flows, we will discuss later.<\/p>\n

  ##### 1.2.3 Join operator<\/h5>\n

  Join establishes a link between two or more columns in a join condition. Strictly speaking, this relationship is not a data flow type.<\/p>\n

  <\/p>\n

  A join relationship will be created after parsing the above SQL. It represents the connection relationship between tbl, between tbl.key and TT.key.<\/p>\n

  <\/p>\n

  ### 2. Entities in the data flow<\/h3>\n

  When building a data flow between two entities: a source entity and a target entity. They can be column-to-column, table-to-column, or table-to-table.<\/p>\n

  #### 2.1 Column to Column<\/h4>\n

  This is the most common case where both entities in the dataflow are columns.<\/p>\n

  #### 2.2 Table to Column<\/h4>\n

  When we say a table affects the value of a column, we usually mean that the total number of rows in a table affects the value of a column. Typically, this column is derived from the COUNT() function.<\/p>\n

  <\/p>\n

  Table-to-column data flow is represented by using the psedou column of table RelationRows, which will build an indirect data flow from scott, relation row RS-1.num_emp.<\/p>\n

  <\/p>\n

  <\/p>\n

  #### 2.3 Table to table<\/h4>\n

  Sometimes, there will be data flow between two tables. For example, in an alter table rename SQL statement, a table-to-table data flow will be created. In effect, table-to-table data flow is determined by using the RelationRows table pseudo column.<\/p>\n

  <\/p>\n

  <\/p>\n

  ### 3. Data lineage<\/h3>\n

  Data lineage consists of many basic data flow units.<\/span><\/p>\n

  <\/p>\n

  Data lineage analysis diagram:<\/p>\n

  Gudu SQLFlow<\/figcaption><\/figure>\n

  The output can also be in XML or JSON format.<\/p>\n

  ### Conclusion<\/h3>\n

  Thank you for reading our article and we hope it can help you to have a better understanding of\u00a0Gudu SQLFlow. If you want to know more about Gudu SQLFlow, we would like to advise you to visit\u00a0Gudu SQLFlow<\/a>\u00a0<\/strong>for more information.\u00a0<\/strong>With the powerful SQL processing capability of the self-developed general-purpose SQL parser, Gudu SQLFlow is your first choice for analyzing SQL data lineage.<\/p>\n","protected":false},"excerpt":{"rendered":"

  What’s Gudu SQLFlow? In large data warehouses and data lakes, complete data lineage can be used for data traceability, impact analysis of table and column changes, proof of data compliance, data quality checks, and more.\u00a0So what exactly is data lineage?\u00a0According to Wikipedia, data lineage includes where the data came from, what happened, and where it moved over time.\u00a0\u00a0Gudu SQLFlow gives complete data lineage by analyzing SQL scripts.\u00a0It can handle SQL statements of more than 20 kinds of databases, including complex stored procedures and dynamic SQL statements.\u00a0With the powerful SQL processing capability of the self-developed general-purpose SQL parser, Gudu SQLFlow is\u2026<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[8],"tags":[5,29],"blocksy_meta":{"styles_descriptor":{"styles":{"desktop":"","tablet":"","mobile":""},"google_fonts":[],"version":5}},"yoast_head":"\nGudu SQLFlow : What's it and why it is important?<\/title>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\t\n\t\n\t\n