Magic ETL on BigQuery

‍

Magic ETL Pushdown for BigQuery

We are thrilled to announce that Magic ETL Pushdown for Google BigQuery is now Beta. While many of you have used Magic ETL to move and transform data within Domo, Pushdown changes the game by allowing you to build workflows visually while executing the processing power directly within your BigQuery environment.

This means your data stays put, your security remains tight, and your compute costs are optimized. To celebrate this milestone, we’ve added two major enhancements to the BigQuery Pushdown experience:

1) Subset Processing Support

We’ve enhanced MagicETL to give you more flexibility and control over your BigQuery outputs. Previously limited to the Replace method, you can now choose from Append, Upsert, and Partition options to better align with your unique dataflow needs.

• Daily updates? Append just one day’s data to your sales table instead of overwriting the entire data set. ‍
• Working with large datasets? Partition your data by date to quickly query specific time periods, like last week’s website traffic, without scanning the entire data set. ‍
• Managing historical snapshots? Use Upsert or Partitioning for smarter, more efficient updates.

This update empowers you to optimize your workflows, maintain historical data, and improve query performance, all while reducing costs. Subset Processing Support is now available in beta in MagicETL output tiles for BigQuery compute integrations.

2) Native SQL Syntax

Advanced data transformation just got more flexible with the addition of Native SQL Syntax in Magic ETL’s SQL tile. This new feature is designed to meet the needs of advanced BigQuery users who prefer using native SQL or already have SQL scripts tailored for their unique use cases.

With Native SQL Syntax, you can now apply BigQuery-native SQL syntax directly within Magic ETL to transform your data without needing to refactor your existing SQL code. Whether you face constraints in your BigQuery environment or want to maximize efficiency, this feature offers a seamless way to integrate and execute your custom transformations.

The Pass-Through SQL option will be available within the SQL tile in Magic ETL, located on the left-hand rail in the Magic ETL authoring canvas. This update is being made available in beta as part of Magic ETL on BigQuery.  

We’re excited to offer this added flexibility for BigQuery users and look forward to seeing how it will enhance your workflows.