[ad_1]
A free various to dbt cloud is Mage, an open-source information pipeline software for information transformation and integration duties.
Mage seamlessly enhances dbt with a spread of advantages, together with:
- Built-in Net-based IDE: Mage supplies a handy web-based IDE the place you’ll be able to develop and discover information fashions effortlessly inside a single interface.
- Language Flexibility: With Mage, you’ll be able to mix the strengths of various instruments and languages alongside dbt for enhanced information processing capabilities.
- Visualizing dbt Mannequin Output: Mage supplies a built-in visualization functionality, permitting customers to effortlessly visualize the output generated by dbt fashions with only a few clicks.
- Information Extraction and Loading: Along with information transformation, Mage presents functionalities for information extraction and loading, enabling a extra complete end-to-end information pipeline answer.
- Pipeline Scheduling and Retry Mechanism: Mage permits you to schedule your information pipelines and robotically retry failed parts, guaranteeing the sleek and dependable execution of your information integration processes.
Let’s dive deeper into every of those options.
Be at liberty to discover and experiment with the supply code by cloning this GitHub repository:
Set up Mage
You possibly can set up Mage utilizing Docker, pip, or conda. This text will use Docker to put in Mage and initialize the mission.
docker run -it -p 6789:6789 -v $(pwd):/house/src mageai/mageai /app/run_app.sh mage begin [project_name]
For instance, let’s title our mission “dbt_mage,” so the command turns into:
docker run -it -p 6789:6789 -v $(pwd):/house/src mageai/mageai /app/run_app.sh mage begin dbt_mage
Discover different methods to put in Mage right here.
Create a pipeline
Open http://localhost:6789/ in your browser to view the Mage UI.
Click on on “New” and choose “Commonplace (batch)” to create a brand new batch pipeline. Rename it as “dbt_pipeline.”
Set up dependencies
Since we are going to use BigQuery as the info warehouse for dbt, we have to set up dbt-bigquery by including it to the “necessities.txt” file and clicking on “Set up packages.”
Create a dbt mission
To create a dbt mission, navigate to the proper panel and click on on the terminal button.
Transfer to the “dbt” folder below your mission and execute the command dbt init:
cd dbt_mage/dbt
dbt init demo -s
This command provides the “demo” folder to the dbt listing.
Proper-click on the “demo” folder and create a brand new file named “profiles.yml.” Specify your BigQuery credentials on this file.
[ad_2]