{"id":2875,"date":"2023-07-12T05:30:41","date_gmt":"2023-07-12T10:30:41","guid":{"rendered":"https:\/\/www.dpriver.com\/blog\/?p=2875"},"modified":"2023-07-12T05:30:42","modified_gmt":"2023-07-12T10:30:42","slug":"ingest-postgresql-metadata-on-datahub","status":"publish","type":"post","link":"https:\/\/www.dpriver.com\/blog\/2023\/07\/ingest-postgresql-metadata-on-datahub\/","title":{"rendered":"Ingest PostgreSQL metadata on Datahub"},"content":{"rendered":"\n

Datahub<\/a> is an open-source metadata platform for the modern data stack. We have integrated the SQLFlow into Datahub so that the SQLFlow data lineage is enabled in the Datahub UI. Check this blog<\/a> for how to integrate SQLFlow into Datahub.<\/p>\n\n\n\n

There are three ways to ingest your PostgreSQL metadata to Datahub:<\/p>\n\n\n\n

- Datahub UI (v0.10.4 is used in this blog)<\/li>
- Command Line<\/li>
- Rest API (Java\/Python sdk are available. This approach is ideal for metadata without a fixed format or any other sources not supported by DataHub)<\/li><\/ul>\n\n\n\n

  This blog will introduce how to ingest the PostgreSQL metadata to Datahub with the first two approaches.<\/p>\n\n\n\n

  ## Prerequisites<\/h2>\n\n\n\n

  Install the following plugins on the Datahub server:<\/p>\n\n\n\n

  ##

  ##

      pip3 install 'acryl-datahub[datahub-rest]' \npip3 install 'acryl-datahub[postgres]'<\/code><\/pre>\n\n\n\nIngest metadata from Datahub UI<\/h2>\n\n\n\na. Click\u00a0Ingestion<\/code>\u00a0on the DataHub menu bar <\/p>\n\n\n\n<\/figure>\n\n\n\nb. Select Sources<\/code> and click Create new source<\/code><\/p>\n\n\n\n<\/figure>\n\n\n\nc. Choose PostgreSQL in the source list
      <\/p>\n\n\n\n<\/figure>\n\n\n\nd. Configure the database connection<\/p>\n\n\n\n<\/figure>\n\n\n\ne. Click\u00a0next<\/code>\u00a0to configure the schedule Ingestion. If you need to update metadata periodically, you can configure schedule Ingestion. It can greatly reduce the workload of manual import.<\/p>\n\n\n\n<\/figure>\n\n\n\nf. Give a name to the source
      <\/p>\n\n\n\n<\/figure>\n\n\n\ng. The source is created when you click\u00a0Save<\/code>. Click\u00a0Edit<\/code>\u00a0to change the configuration and\u00a0EXECUTE<\/code>\u00a0to run the ingestion.
      <\/p>\n\n\n\n<\/figure>\n\n\n\nh. After clicking\u00a0EXECUTE<\/code>, the task will be executed immediately. if a schedule task is configured, the import will be automatically triggered when the scheduling condition is met.\u00a0Last Status<\/code>\u00a0will show the execution status of the last ingestion.
      <\/p>\n\n\n\n<\/figure>\n\n\n\ni. Click the\u00a0+<\/code>\u00a0at the beginning of each source to expand and view the execution records corresponding to the source. If the status is\u00a0Failed<\/code>, you can check the logs with the\u00a0DETAILS<\/code>\u00a0button.
      <\/p>\n\n\n\n<\/figure>\n\n\n\nWhen Status is\u00a0Succeeded<\/code>, you will have the MySQL icon in the Datahub home page.<\/p>\n\n\n\n<\/figure>\n\n\n\nIngest via Command Line<\/h2>\n\n\n\na. Edit the config file and upload that to your datahub server<\/p>\n\n\n\nCheck https:\/\/datahubproject.io\/docs\/generated\/ingestion\/sources\/mysql\/#cli-based-ingestion for the config structure.
      <\/p>\n\n\n\n<\/figure>\n\n\n\nb. Run the following command to read the config file<\/p>\n\n\n\ndatahub ingest -c \/home\/datahub\/PostgreSQL.yaml<\/code><\/pre>\n\n\n\n
      <\/p>\n","protected":false},"excerpt":{"rendered":"Datahub is an open-source metadata platform for the modern data stack. We have integrated the SQLFlow into Datahub so that the SQLFlow data lineage is enabled in the Datahub UI. Check this blog for how to integrate SQLFlow into Datahub. There are three ways to ingest your PostgreSQL metadata to Datahub: Datahub UI (v0.10.4 is used in this blog) Command Line Rest API (Java\/Python sdk are available. This approach is ideal for metadata without a fixed format or any other sources not supported by DataHub) This blog will introduce how to ingest the PostgreSQL metadata to Datahub with the first two approaches. Prerequisites\u2026<\/p>\n","protected":false},"author":6,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"blocksy_meta":{"styles_descriptor":{"styles":{"desktop":"","tablet":"","mobile":""},"google_fonts":[],"version":5}},"yoast_head":"\nIngest PostgreSQL metadata on Datahub<\/title>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\t\n\t\n\t\n