{"id":2649,"date":"2023-02-10T10:15:36","date_gmt":"2023-02-10T15:15:36","guid":{"rendered":"https:\/\/www.dpriver.com\/blog\/?p=2649"},"modified":"2023-02-23T10:07:52","modified_gmt":"2023-02-23T15:07:52","slug":"generate-data-lineage-with-sqlflow-ingester","status":"publish","type":"post","link":"https:\/\/www.dpriver.com\/blog\/2023\/02\/generate-data-lineage-with-sqlflow-ingester\/","title":{"rendered":"Generate data lineage with SQLFlow-Ingester"},"content":{"rendered":"\n<h3 class=\"wp-block-heading\">Why SQLFlow-Ingester?<\/h3>\n\n\n\n<p><a href=\"https:\/\/www.dpriver.com\/blog\/2022\/08\/20\/introduction-of-gudu-sqlflow\/\">Gudu SQLFlow<\/a> supports two kinds of file as the input:<\/p>\n\n\n\n<ul><li>SQL file, including comments. DDL file for an example, sql such as create table can be used as metadata.<\/li><\/ul>\n\n\n\n<ul><li>Json files which contain the database metadata<\/li><\/ul>\n\n\n\n<p>All other kinds of the input will be no longer supported and users shall convert the inputs to the above two formats. As a result, an out-of-box tool is needed to complete that conversion and automatically submit SQLFlow task.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">SQLFlow-Ingester Basic<\/h3>\n\n\n\n<p>SQLFlow-Ingester is a tool that helps you to extract metadata from various databases and create SQLFlow job based on your database. The extracted metadata file can also be used by <a href=\"https:\/\/www.gudusoft.com\/sqlflow-java-library-2\/\" target=\"_blank\" rel=\"noreferrer noopener\">dlineage tool<\/a> to generate data lineage.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" loading=\"lazy\" width=\"456\" height=\"511\" src=\"https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2023\/02\/\u56fe\u7247.png\" alt=\"\" class=\"wp-image-2651\" srcset=\"https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2023\/02\/\u56fe\u7247.png 456w, https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2023\/02\/\u56fe\u7247-268x300.png 268w\" sizes=\"(max-width: 456px) 100vw, 456px\" \/><\/figure>\n\n\n\n<p>SQLFlow-Ingester download address: <a href=\"https:\/\/github.com\/sqlparser\/sqlflow_public\/releases\">https:\/\/github.com\/sqlparser\/sqlflow_public\/releases<\/a><\/p>\n\n\n\n<p>SQLFlow-Ingester has three different parts:<\/p>\n\n\n\n<ul><li>sqlflow-exporter: getting metadata from database<\/li><li>sqlflow-extractor: processing raw data files such as log files, various script files (from which SQL statements and metadata to be processed are extracted), CSV files containing SQL statements, etc.<\/li><li>sqlflow-submitter: submitting sql and metadata to the sqlflow server, creating jobs, generating data lineage, and having the results in the UI.<\/li><\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">SQLFlow-Exporter<\/h3>\n\n\n\n<p>SQLFlow-exporter is the main functionality of the SQLFlow-Ingester. It can be used to get the metadata from different databases.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">SQLFlow-Exporter Usage<\/h4>\n\n\n\n<p><strong>Under Linux &amp; Mac<\/strong>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>.\/exporter.sh -host 127.0.0.1 -port 1521 -db orcl -user scott -pwd tiger -save \/tmp\/sqlflow-ingester -dbVendor dbvoracle<\/code><\/pre>\n\n\n\n<p><strong>Under windows<\/strong>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>exporter.bat -host 127.0.0.1 -port 1521 -db orcl -user scott -pwd tiger -save c:\\tmp\\sqlflow-ingester -dbVendor dbvoracle<\/code><\/pre>\n\n\n\n<p>Or you can directly execute the .jar package, the jar packages are under the <code>.\/lib<\/code> folder:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>java -jar sqlflow-exporter-1.0.jar  -host 106.54.xx.xx -port 1521 -db orcl -user username -pwd password -save d:\/ -dbVendor dbvoracle<\/code><\/pre>\n\n\n\n<p>A metadata.json file will be generated after the success export of metadta from the database.<\/p>\n\n\n\n<p>Example for the success output:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>exporter metadata success: &lt;-save&gt;\/metadata.json<\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" loading=\"lazy\" width=\"761\" height=\"29\" src=\"https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2023\/02\/\u56fe\u7247-1.png\" alt=\"\" class=\"wp-image-2653\" srcset=\"https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2023\/02\/\u56fe\u7247-1.png 761w, https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2023\/02\/\u56fe\u7247-1-300x11.png 300w\" sizes=\"(max-width: 761px) 100vw, 761px\" \/><\/figure>\n\n\n\n<p><strong>Parameters<\/strong><br>-dbVendor: Database type. Use colon to split dbVendor and version if specific version is required. (:, such as dbvmysql:5.7)<br>-host: Database host name (ip address or domain name)<br>-port: Port number<br>-db: Database name<br>-user: User name<br>-pwd: User password<br>-save: Destination folder path where we put the exported metadata json file. The exported file will be in name as metadata.json.<br>-extractedDbsSchemas: Export metadata under the specific schema. Use comma to split if multiple schema required (such as ,). We can use this flag to improve the export performance.<br>-excludedDbsSchemas: Exclude metadata under the specific schema during the export. Use comma to split if multiple schema required (such as ,). We can use this flag to improve the export performance.<br>-extractedViews: Export metadata under the specific view. Use comma to split if multiple views required (such as ,). We can use this flag to improve the export performance.<br>-merge: Merge the metadata results which are exported in different process. Use comma to split files to merge.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Improve Export Performance<\/h4>\n\n\n\n<p>Time consumed during the export from database could be very long if the data volume is huge. Following actions are made to improve the Ingester export performance:<\/p>\n\n\n\n<ul><li>JDBC <code>fetchsize<\/code> is set to 1000 for table and view. For sql of view and process the <code>fetchsize<\/code> is set to 500. (check this oracle doc for what is jdbc <code>fetchsize<\/code>: <a href=\"https:\/\/docs.oracle.com\/middleware\/1212\/toplink\/TLJPA\/q_jdbc_fetch_size.htm#TLJPA647\">https:\/\/docs.oracle.com\/middleware\/1212\/toplink\/TLJPA\/q_jdbc_fetch_size.htm#TLJPA647<\/a>)<\/li><li>Parameters <code>extractedDbsSchemas<\/code>, <code>excludedDbsSchemas<\/code> and <code>extractedViews<\/code> are improved for oracle and postgresql. A plan has already been made to improve the implementation of these parameters for other databases.<\/li><\/ul>\n\n\n\n<p><code>-merge<\/code> can be used to merge the metadata results. Use comma to split files to merge. All files under the folder will be merged if the given value is the folder path.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>exporter.bat -dbVendor dbvpostgresql -extractedDbsSchemas kingland.dbt%,kingland.pub%  -host 115.159.225.38 -port 5432 -db kingland -user bigking -pwd Cat_*** -save D:\/out\/1.json\nexporter.bat -dbVendor dbvpostgresql -extractedDbsSchemas kingland.sqlflow  -host 115.159.225.38 -port 5432 -db kingland -user bigking -pwd Cat_*** -save D:\/out\/2.json\nexporter.bat -merge  D:\/out\/1.json,D:\/out\/2.json    -save D:\/out\/merge.json\nor\nexporter.bat -merge  D:\/out\/   -save D:\/out\/merge.json<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">SQLFlow-Submitter<\/h3>\n\n\n\n<p>After decompressing the package, you will find <code>submitter.bat<\/code> for Windows and <code>submitter.sh<\/code> for Linux &amp; Mac.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" loading=\"lazy\" width=\"640\" height=\"210\" src=\"https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2023\/02\/\u56fe\u7247-2.png\" alt=\"\" class=\"wp-image-2656\" srcset=\"https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2023\/02\/\u56fe\u7247-2.png 640w, https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2023\/02\/\u56fe\u7247-2-300x98.png 300w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/><figcaption><strong>Under Linux &amp; Mac<\/strong>:<\/figcaption><\/figure>\n\n\n\n<pre class=\"wp-block-code\"><code>bash submitter.sh -f &lt;path_to_config_file&gt;  \n\nnote: \n    path_to_config_file: the full path to the config file\n\neg: \n    bash submitter.sh -f \/home\/workspace\/gudu_ingester\/submitter_config.json<\/code><\/pre>\n\n\n\n<p><strong>Under Windows<\/strong>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>submitter.bat -f D:\/mssql-winuser-config.json<\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" loading=\"lazy\" width=\"1024\" height=\"83\" src=\"https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2023\/02\/\u56fe\u7247-3-1024x83.png\" alt=\"\" class=\"wp-image-2657\" srcset=\"https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2023\/02\/\u56fe\u7247-3-1024x83.png 1024w, https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2023\/02\/\u56fe\u7247-3-300x24.png 300w, https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2023\/02\/\u56fe\u7247-3-768x62.png 768w, https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2023\/02\/\u56fe\u7247-3.png 1271w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>After successfully executing the Submitter, check the Job list on UI and you will find the job submitted by the Submitter, with the Job name configured in the configuration file.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Cron Job<a href=\"https:\/\/app.gitbook.com\/o\/8tYHSVB0mG81POwyVWfj\/s\/HPaNIbjpxIoccnaNkyJQ\/6.-sqlflow-ingester\/introduction\/sqlflow-submitter#cron-job\"><\/a><\/h4>\n\n\n\n<p>If you are on Linux or Mac, you can schedule the submitter with <code>crontab<\/code> to create a cron job.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>crontab -e<\/code><\/pre>\n\n\n\n<p>In the editor opened by the above command, let&#8217;s say now we want to schedule a daily cron job, add the following code:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>0 0 * * * bash submitter.sh -f &lt;path_to_config_file&gt; &lt;lib_path&gt;\n\nnote: \n    path_to_config_file: config file path \n    lib_path: lib directory absolute path<\/code><\/pre>\n\n\n\n<p>Please check <a href=\"https:\/\/phoenixnap.com\/kb\/set-up-cron-job-linux\">this document<\/a> for more information about <code>cron<\/code> and you can learn the <code>cron<\/code> grammar with <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cron\">Wikipedia<\/a>.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">SQLFlow-Submitter Log<a href=\"https:\/\/app.gitbook.com\/o\/8tYHSVB0mG81POwyVWfj\/s\/HPaNIbjpxIoccnaNkyJQ\/6.-sqlflow-ingester\/introduction\/sqlflow-submitter#sqlflow-submitter-log\"><\/a><\/h4>\n\n\n\n<p>The logs of the submitter are persisted under <code>log<\/code> folder.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" loading=\"lazy\" width=\"1024\" height=\"397\" src=\"https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2023\/02\/\u56fe\u7247-4-1024x397.png\" alt=\"\" class=\"wp-image-2658\" srcset=\"https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2023\/02\/\u56fe\u7247-4-1024x397.png 1024w, https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2023\/02\/\u56fe\u7247-4-300x116.png 300w, https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2023\/02\/\u56fe\u7247-4-768x297.png 768w, https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2023\/02\/\u56fe\u7247-4.png 1268w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>Why SQLFlow-Ingester? Gudu SQLFlow supports two kinds of file as the input: SQL file, including comments. DDL file for an example, sql such as create table can be used as metadata. Json files which contain the database metadata All other kinds of the input will be no longer supported and users shall convert the inputs to the above two formats. As a result, an out-of-box tool is needed to complete that conversion and automatically submit SQLFlow task. SQLFlow-Ingester Basic SQLFlow-Ingester is a tool that helps you to extract metadata from various databases and create SQLFlow job based on your database.\u2026<\/p>\n","protected":false},"author":6,"featured_media":2436,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[66,93],"tags":[],"blocksy_meta":{"styles_descriptor":{"styles":{"desktop":"","tablet":"","mobile":""},"google_fonts":[],"version":5}},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v19.4 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Generate data lineage with SQLFlow-Ingester<\/title>\n<meta name=\"description\" content=\"Generate data lineage with SQLFlow-Ingester sqlflow-ingester\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dpriver.com\/blog\/2023\/02\/generate-data-lineage-with-sqlflow-ingester\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Generate data lineage with SQLFlow-Ingester\" \/>\n<meta property=\"og:description\" content=\"Generate data lineage with SQLFlow-Ingester sqlflow-ingester\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dpriver.com\/blog\/2023\/02\/generate-data-lineage-with-sqlflow-ingester\/\" \/>\n<meta property=\"og:site_name\" content=\"SQL and Data Blog\" \/>\n<meta property=\"article:published_time\" content=\"2023-02-10T15:15:36+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-02-23T15:07:52+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2022\/08\/Introduction_of_Gudu_SQLFlow-3-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"672\" \/>\n\t<meta property=\"og:image:height\" content=\"357\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"leo gu\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"leo gu\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.dpriver.com\/blog\/#organization\",\"name\":\"SQL and Data Blog\",\"url\":\"https:\/\/www.dpriver.com\/blog\/\",\"sameAs\":[],\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.dpriver.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2022\/07\/sqlpp-character.png\",\"contentUrl\":\"https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2022\/07\/sqlpp-character.png\",\"width\":251,\"height\":72,\"caption\":\"SQL and Data Blog\"},\"image\":{\"@id\":\"https:\/\/www.dpriver.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dpriver.com\/blog\/#website\",\"url\":\"https:\/\/www.dpriver.com\/blog\/\",\"name\":\"SQL and Data Blog\",\"description\":\"SQL related blog for database professional\",\"publisher\":{\"@id\":\"https:\/\/www.dpriver.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dpriver.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dpriver.com\/blog\/2023\/02\/generate-data-lineage-with-sqlflow-ingester\/\",\"url\":\"https:\/\/www.dpriver.com\/blog\/2023\/02\/generate-data-lineage-with-sqlflow-ingester\/\",\"name\":\"Generate data lineage with SQLFlow-Ingester\",\"isPartOf\":{\"@id\":\"https:\/\/www.dpriver.com\/blog\/#website\"},\"datePublished\":\"2023-02-10T15:15:36+00:00\",\"dateModified\":\"2023-02-23T15:07:52+00:00\",\"description\":\"Generate data lineage with SQLFlow-Ingester sqlflow-ingester\",\"breadcrumb\":{\"@id\":\"https:\/\/www.dpriver.com\/blog\/2023\/02\/generate-data-lineage-with-sqlflow-ingester\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dpriver.com\/blog\/2023\/02\/generate-data-lineage-with-sqlflow-ingester\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dpriver.com\/blog\/2023\/02\/generate-data-lineage-with-sqlflow-ingester\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.dpriver.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Generate data lineage with SQLFlow-Ingester\"}]},{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dpriver.com\/blog\/2023\/02\/generate-data-lineage-with-sqlflow-ingester\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dpriver.com\/blog\/2023\/02\/generate-data-lineage-with-sqlflow-ingester\/\"},\"author\":{\"name\":\"leo gu\",\"@id\":\"https:\/\/www.dpriver.com\/blog\/#\/schema\/person\/9e80d249b6da4e572bd8590b0789bc14\"},\"headline\":\"Generate data lineage with SQLFlow-Ingester\",\"datePublished\":\"2023-02-10T15:15:36+00:00\",\"dateModified\":\"2023-02-23T15:07:52+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dpriver.com\/blog\/2023\/02\/generate-data-lineage-with-sqlflow-ingester\/\"},\"wordCount\":657,\"publisher\":{\"@id\":\"https:\/\/www.dpriver.com\/blog\/#organization\"},\"articleSection\":[\"Data Governance\",\"SQLFlow\"],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dpriver.com\/blog\/#\/schema\/person\/9e80d249b6da4e572bd8590b0789bc14\",\"name\":\"leo gu\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.dpriver.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/24976e2e4ca7dd476652bb26bd09392b?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/24976e2e4ca7dd476652bb26bd09392b?s=96&d=mm&r=g\",\"caption\":\"leo gu\"},\"url\":\"https:\/\/www.dpriver.com\/blog\/author\/guyuanhao\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Generate data lineage with SQLFlow-Ingester","description":"Generate data lineage with SQLFlow-Ingester sqlflow-ingester","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dpriver.com\/blog\/2023\/02\/generate-data-lineage-with-sqlflow-ingester\/","og_locale":"en_US","og_type":"article","og_title":"Generate data lineage with SQLFlow-Ingester","og_description":"Generate data lineage with SQLFlow-Ingester sqlflow-ingester","og_url":"https:\/\/www.dpriver.com\/blog\/2023\/02\/generate-data-lineage-with-sqlflow-ingester\/","og_site_name":"SQL and Data Blog","article_published_time":"2023-02-10T15:15:36+00:00","article_modified_time":"2023-02-23T15:07:52+00:00","og_image":[{"width":672,"height":357,"url":"https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2022\/08\/Introduction_of_Gudu_SQLFlow-3-1.png","type":"image\/png"}],"author":"leo gu","twitter_card":"summary_large_image","twitter_misc":{"Written by":"leo gu","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Organization","@id":"https:\/\/www.dpriver.com\/blog\/#organization","name":"SQL and Data Blog","url":"https:\/\/www.dpriver.com\/blog\/","sameAs":[],"logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.dpriver.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2022\/07\/sqlpp-character.png","contentUrl":"https:\/\/www.dpriver.com\/blog\/wp-content\/uploads\/2022\/07\/sqlpp-character.png","width":251,"height":72,"caption":"SQL and Data Blog"},"image":{"@id":"https:\/\/www.dpriver.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"WebSite","@id":"https:\/\/www.dpriver.com\/blog\/#website","url":"https:\/\/www.dpriver.com\/blog\/","name":"SQL and Data Blog","description":"SQL related blog for database professional","publisher":{"@id":"https:\/\/www.dpriver.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dpriver.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.dpriver.com\/blog\/2023\/02\/generate-data-lineage-with-sqlflow-ingester\/","url":"https:\/\/www.dpriver.com\/blog\/2023\/02\/generate-data-lineage-with-sqlflow-ingester\/","name":"Generate data lineage with SQLFlow-Ingester","isPartOf":{"@id":"https:\/\/www.dpriver.com\/blog\/#website"},"datePublished":"2023-02-10T15:15:36+00:00","dateModified":"2023-02-23T15:07:52+00:00","description":"Generate data lineage with SQLFlow-Ingester sqlflow-ingester","breadcrumb":{"@id":"https:\/\/www.dpriver.com\/blog\/2023\/02\/generate-data-lineage-with-sqlflow-ingester\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dpriver.com\/blog\/2023\/02\/generate-data-lineage-with-sqlflow-ingester\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dpriver.com\/blog\/2023\/02\/generate-data-lineage-with-sqlflow-ingester\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.dpriver.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Generate data lineage with SQLFlow-Ingester"}]},{"@type":"Article","@id":"https:\/\/www.dpriver.com\/blog\/2023\/02\/generate-data-lineage-with-sqlflow-ingester\/#article","isPartOf":{"@id":"https:\/\/www.dpriver.com\/blog\/2023\/02\/generate-data-lineage-with-sqlflow-ingester\/"},"author":{"name":"leo gu","@id":"https:\/\/www.dpriver.com\/blog\/#\/schema\/person\/9e80d249b6da4e572bd8590b0789bc14"},"headline":"Generate data lineage with SQLFlow-Ingester","datePublished":"2023-02-10T15:15:36+00:00","dateModified":"2023-02-23T15:07:52+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dpriver.com\/blog\/2023\/02\/generate-data-lineage-with-sqlflow-ingester\/"},"wordCount":657,"publisher":{"@id":"https:\/\/www.dpriver.com\/blog\/#organization"},"articleSection":["Data Governance","SQLFlow"],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dpriver.com\/blog\/#\/schema\/person\/9e80d249b6da4e572bd8590b0789bc14","name":"leo gu","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.dpriver.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/24976e2e4ca7dd476652bb26bd09392b?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/24976e2e4ca7dd476652bb26bd09392b?s=96&d=mm&r=g","caption":"leo gu"},"url":"https:\/\/www.dpriver.com\/blog\/author\/guyuanhao\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dpriver.com\/blog\/wp-json\/wp\/v2\/posts\/2649"}],"collection":[{"href":"https:\/\/www.dpriver.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dpriver.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dpriver.com\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dpriver.com\/blog\/wp-json\/wp\/v2\/comments?post=2649"}],"version-history":[{"count":5,"href":"https:\/\/www.dpriver.com\/blog\/wp-json\/wp\/v2\/posts\/2649\/revisions"}],"predecessor-version":[{"id":2659,"href":"https:\/\/www.dpriver.com\/blog\/wp-json\/wp\/v2\/posts\/2649\/revisions\/2659"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.dpriver.com\/blog\/wp-json\/wp\/v2\/media\/2436"}],"wp:attachment":[{"href":"https:\/\/www.dpriver.com\/blog\/wp-json\/wp\/v2\/media?parent=2649"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dpriver.com\/blog\/wp-json\/wp\/v2\/categories?post=2649"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dpriver.com\/blog\/wp-json\/wp\/v2\/tags?post=2649"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}