​

Intro

Creating a Magic ETL DataFlow is simple with our intuitive drag-and-drop interface. Drag an input DataSet onto the canvas, then add tiles to specify how the DataSets should be joined and transformed. Tiles allow you to aggregate, combine data, perform data science operations, manipulate DataSets, work with dates and numbers, filter, configure column performance, pivot, scripting, transform text, and perform a number of utility tasks such as adding constants and duplicating columns. This article describes the basic steps to create a DataFlow in Magic ETL, including how to work with Personalized Data Permissions (PDP) and best practices.

---

• Magic ETL DataFlows and PDP
• Magic ETL canvas tools
  • Coarser grid
  • Mini map
  • Tile grouping
  • Tile support
  • Undo/redo
  • Use sections
  • Use sticky notes
  • Use column search
• Create a Magic ETL DataFlow
• Best practices for Magic ETL DataFlows
• Recover unsaved changes

---

​

Magic ETL DataFlows and PDP

​

Magic ETL Canvas Tools

The Magic ETL canvas includes many tools to help you quickly create your DataFlow.

Coarser Grid

By default, the Magic ETL canvas has a fine-grained snap-to-grid of 12 pixels. You can modify this and choose a coarser snap-to-grid functionality of 32 pixels (half the size of a Magic ETL tile). To enable the coarse grid, select Settings in the canvas toolbar to open the DataFlow Settings modal. Note: Changes to DataFlow Settings apply only to this DataFlow and do not persist across other DataFlows. Go to the Canvas Settings tab in the modal and check the box for Larger Grid. Choose the grid line style from the dropdown.

Mini Map

The mini map displays in the corner of the screen and helps you see the layout and navigate around complex and detailed DataFlows. Click and drag the white square in the mini map to move to a certain view of the DataFlow on the canvas. If you close the mini map, reopen it by selecting the map pointer icon in the same corner where it appears.

Tile Grouping

You can also select a number of tiles at once by clicking on the canvas and then dragging the cursor over them. When multiple tiles are selected, you can drag all of the selected tiles as a group to where you want them. You can also delete the selected tiles by selecting Delete in the left panel.

Tile Support

You can get help while configuring a tile on the canvas by selecting the tile and then selecting Help (question mark icon).

Undo/Redo

You can undo/redo actions taken while creating a Magic ETL DataFlow. The controls (arrow icons) are found in the toolbar at the top of the canvas. You can also use the dropdown to view an action history list of all actions taken during a given editing session. Using the list, you can jump forward and backward in time to see the DataFlow in that state.

​

Use Sections

Sections can help you organize your canvas by logically and visually grouping tiles together. Add one or more sections to your canvas by following the steps below:

1. Highlight the tile(s) you want to include in a section and right-click to display the section options.
2. Select Create Section.
   
3. After creating a section, you can use the color picker to choose a background color.
   
4. (Optional) Rename a section by selecting More (three vertical dots icon) > Rename Section.
   
5. (Optional) Resize a section by clicking and dragging its edges or corners. Note: A section will pick up tiles and notes that are overlapped when resizing.
6. (Optional) Group a sticky note with a section by dropping it into the section or dragging the edge of the section over the sticky note.

​

Use Sticky Notes

Sticky notes help you document and annotate so you can communicate to a DataFlow viewer about the DataFlow’s current status, history, and work that may need to be done.

1. Right-click anywhere on the Magic ETL canvas and choose Add Comment.
   
   A sticky note appears with an editing bar along the top.
2. Use the editing bar to choose a note color and other rich text options, including hyperlinking.
   
   Tip: You can stamp your signature on a sticky note so people can identify the author.

   
3. (Optional) You can group a sticky note with a section by dragging it at least halfway onto the section.
   

​

Use Column Search

Column Search in Magic ETL allows you to view all tiles in your DataFlow that contain a certain column. Follow these steps to use Column Search:

1. While creating your DataFlow, select the magnifying glass in the utility bar above the canvas to expand the search field.

2. Enter a partial or complete column name in the search field. All entered characters are included in the search.

3. Observe the orange-highlighted tiles (and relevant paths) on the canvas. Each highlighted tile includes a column that matches your search term(s).

​

Create a Magic ETL DataFlow

Follow these steps to create a Magic ETL DataFlow:

1. Navigate to the Domo Data Center (More > Data).
2. If you aren’t already there, use the left sidebar to go to the Data Warehouse.
3. In the ribbon at the top of the Data Center, select Transform Data > Magic ETL to open the Magic ETL canvas.
   
4. In the left panel, expand DataSets and drag an Input DataSet tile to the canvas.
   
   The tile editor expands below the canvas.
   
5. In the tile editor, select Choose DataSet to choose the DataSet you want to transform.
6. Drag other tiles to the canvas, depending on what transformations you want to make, and make sure they are all connected by dragging the nodes on the sides of each tile to the node on the next tile.

7. After configuring all your other tiles, add an output DataSet by dragging an Output DataSet tile from the DataSets tiles to the canvas and connecting it to the last configured tile.
8. Configure the Output DataSet tile by doing the following:
   1. Connect a tile to the Output DataSet tile.
   2. Select the Output DataSet tile, then enter a name for the new output DataSet.
9. (Optional) Schedule the DataFlow. By default, you must run the DataFlow manually. You can schedule it to run when a trigger activates. See Advanced DataFlow Triggering to learn more.
10. Enter a name and description for the DataFlow.
11. Select Save to keep your changes, adding an optional version description before saving again. When you save a DataFlow, an entry for this version is added to the Versions tab in the Details view for the DataFlow. If you add a version description, it appears in the version entry. Learn about Viewing the Version History for a DataFlow.
    • (Conditional) If you choose Save and Run, you can add a description of the DataFlow version and any triggers, then select Save and Run again. The History tab of the DataFlow Details opens, displaying run statistics and other information.
      
      Why are output DataSets not marked as Updated when the DataFlow completes successfully?This is usually because the data has not actually changed—no update has occurred. A DataSet displays as updated if the data has changed during a successful DataFlow execution.

​

Best Practices for Magic ETL DataFlows

We recommend the following for your DataFlow:

• Only include the DataSets that are necessary to create the output DataSet.
• Filter out rows that you don’t need at the beginning of the DataFlow. Learn about Filter tiles.
• Reduce the number of columns to only those you need.
• Use descriptive names for each tile in your DataFlow.
• List the following in your DataFlow description:
  • The input DataSets being transformed and their owner’s names.
  • The output DataSet created.
• Give your DataFlow the same name as the output DataSet. This is because the outputs of a DataFlow become their own DataSet in the Data Center, and this allows you to more easily identify which DataFlows contribute to which output DataSets.
• Be aware that some tiles take longer to execute than others, including:
  • Group By
  • Rank & Window
  • Join Data
  • Remove Duplicates
  • Pivot
  • Scripting tiles
  • Data Science tiles

​

Recover Unsaved Changes

If you close the window without saving your DataFlow, you can recover it by following these steps:

1. Access the DataFlow that was closed without saving. Notice the blue banner at the top of the DataFlow.
   
2. Select Recover changes in the right of the banner. Changes to your DataFlow are recovered.
   Important: If you accidentally close the blue banner without recovering changes, you lose those changes and cannot recover them.