Streamline dbt Mannequin Growth with Pocket book-Type Workspace | by Khuyen Tran | Jun, 2023



A free various to dbt cloud is Mage, an open-source information pipeline software for information transformation and integration duties.

Mage seamlessly enhances dbt with a spread of advantages, together with:

  1. Built-in Net-based IDE: Mage supplies a handy web-based IDE the place you’ll be able to develop and discover information fashions effortlessly inside a single interface.
  2. Language Flexibility: With Mage, you’ll be able to mix the strengths of various instruments and languages alongside dbt for enhanced information processing capabilities.
  3. Visualizing dbt Mannequin Output: Mage supplies a built-in visualization functionality, permitting customers to effortlessly visualize the output generated by dbt fashions with only a few clicks.
  4. Information Extraction and Loading: Along with information transformation, Mage presents functionalities for information extraction and loading, enabling a extra complete end-to-end information pipeline answer.
  5. Pipeline Scheduling and Retry Mechanism: Mage permits you to schedule your information pipelines and robotically retry failed parts, guaranteeing the sleek and dependable execution of your information integration processes.

Let’s dive deeper into every of those options.

Be at liberty to discover and experiment with the supply code by cloning this GitHub repository:

Set up Mage

You possibly can set up Mage utilizing Docker, pip, or conda. This text will use Docker to put in Mage and initialize the mission.

docker run -it -p 6789:6789 -v $(pwd):/house/src mageai/mageai /app/ mage begin [project_name]

For instance, let’s title our mission “dbt_mage,” so the command turns into:

docker run -it -p 6789:6789 -v $(pwd):/house/src mageai/mageai /app/ mage begin dbt_mage

Discover different methods to put in Mage right here.

Create a pipeline

Open http://localhost:6789/ in your browser to view the Mage UI.

Click on on “New” and choose “Commonplace (batch)” to create a brand new batch pipeline. Rename it as “dbt_pipeline.”

Picture by Creator

Set up dependencies

Since we are going to use BigQuery as the info warehouse for dbt, we have to set up dbt-bigquery by including it to the “necessities.txt” file and clicking on “Set up packages.”

Picture by Creator

Create a dbt mission

To create a dbt mission, navigate to the proper panel and click on on the terminal button.

Picture by Creator

Transfer to the “dbt” folder below your mission and execute the command dbt init:

cd dbt_mage/dbt 
dbt init demo -s

This command provides the “demo” folder to the dbt listing.

Picture by Creator

Proper-click on the “demo” folder and create a brand new file named “profiles.yml.” Specify your BigQuery credentials on this file.