{"id":2858,"date":"2023-07-11T09:03:30","date_gmt":"2023-07-11T14:03:30","guid":{"rendered":"https:\/\/www.dpriver.com\/blog\/?p=2858"},"modified":"2023-07-12T05:31:42","modified_gmt":"2023-07-12T10:31:42","slug":"ingest-mysql-metadata","status":"publish","type":"post","link":"https:\/\/www.dpriver.com\/blog\/2023\/07\/ingest-mysql-metadata\/","title":{"rendered":"Ingest MySQL metadata on Datahub"},"content":{"rendered":"\n

<\/p>\n\n\n\n

Datahub<\/a> is an open-source metadata platform for the modern data stack. We have integrated the SQLFlow into Datahub so that the SQLFlow data lineage is enabled in the Datahub UI. Check this blog<\/a> for how to integrate SQLFlow into Datahub.<\/p>\n\n\n\n

There are three ways to ingest your MySQL metadata to Datahub:<\/p>\n\n\n\n

- Datahub UI (v0.10.4 is used in this blog)<\/li>
- Command Line<\/li>
- Rest API (Java\/Python sdk are available. This approach is ideal for metadata without a fixed format or any other sources not supported by DataHub)<\/li><\/ul>\n\n\n\n

  This blog will introduce how to ingest the MySQL metadata to Datahub with the first two approaches.<\/p>\n\n\n\n

  ## Prerequisites<\/h2>\n\n\n\n

  Install the following plugins on the Datahub server:<\/p>\n\n\n\n

  ##

  ##

      pip3 install 'acryl-datahub[datahub-rest]' \npip3 install 'acryl-datahub[mysql]'<\/code><\/pre>\n\n\n\nIngest metadata from Datahub UI<\/h2>\n\n\n\na. Click Ingestion<\/code> on the DataHub menu bar <\/p>\n\n\n\n<\/figure>\n\n\n\nb. Select Sources<\/code> and click Create new source<\/code>
      <\/p>\n\n\n\n<\/figure>\n\n\n\nc. Choose MySQL in the source list
      <\/p>\n\n\n\n<\/figure>\n\n\n\nd. Configure the database connection
      <\/p>\n\n\n\n<\/figure>\n\n\n\ne. Click next<\/code> to configure the schedule Ingestion. If you need to update metadata periodically, you can configure schedule Ingestion. It can greatly reduce the workload of manual import.
      <\/p>\n\n\n\n<\/figure>\n\n\n\nf. Give a name to the source<\/p>\n\n\n\n<\/figure>\n\n\n\ng. The source is created when you click Save<\/code>. Click Edit<\/code> to change the configuration and EXECUTE<\/code> to run the ingestion.<\/p>\n\n\n\n<\/figure>\n\n\n\nh. After clicking EXECUTE<\/code>, the task will be executed immediately. if a schedule task is configured, the import will be automatically triggered when the scheduling condition is met. Last Status<\/code> will show the execution status of the last ingestion.
      <\/p>\n\n\n\n<\/figure>\n\n\n\n i. Click the +<\/code> at the beginning of each source to expand and view the execution records corresponding to the source. If the status is Failed<\/code>, you can check the logs with the DETAILS<\/code> button.
      <\/p>\n\n\n\n
      When Status is Succeeded<\/code>, you will have the MySQL icon in the Datahub home page.<\/figcaption><\/figure>\n\n\n\n<\/figure>\n\n\n\nIngest via Command Line<\/h2>\n\n\n\na. Edit the config file and upload that to your datahub server<\/p>\n\n\n\nCheck https:\/\/datahubproject.io\/docs\/generated\/ingestion\/sources\/mysql\/#cli-based-ingestion for the config structure.<\/p>\n\n\n\n<\/figure>\n\n\n\nb. Run the following command to read the config file<\/p>\n\n\n\ndatahub ingest -c \/home\/datahub\/mysql.yaml<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"Datahub is an open-source metadata platform for the modern data stack. We have integrated the SQLFlow into Datahub so that the SQLFlow data lineage is enabled in the Datahub UI. Check this blog for how to integrate SQLFlow into Datahub. There are three ways to ingest your MySQL metadata to Datahub: Datahub UI (v0.10.4 is used in this blog) Command Line Rest API (Java\/Python sdk are available. This approach is ideal for metadata without a fixed format or any other sources not supported by DataHub) This blog will introduce how to ingest the MySQL metadata to Datahub with the first\u2026<\/p>\n","protected":false},"author":6,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[66],"tags":[],"blocksy_meta":{"styles_descriptor":{"styles":{"desktop":"","tablet":"","mobile":""},"google_fonts":[],"version":5}},"yoast_head":"\nIngest MySQL metadata on Datahub<\/title>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\t\n\t\n\t\n