Data transformation vs. data visualization

The data analysis process has many steps. To get the most use out of their BI tool, businesses need to work through all the steps of the process. Otherwise, their analytics won’t be useful and they won’t be able to use their data to drive insight.

One issue that many businesses have with the data process is that many of the steps seem simple or easily automated, but in reality involve lots of manual, expert-level work. Many of the most important steps, in fact, are the ones that seem simplest on the face.

Some steps, like data transformation, seem hard and ultimately unimportant, so businesses don’t make any effort to do them. Other steps, like data visualization, seem so easy that businesses don’t approach them in a serious way.

Many businesses aren’t able to build an effective data strategy because they don’t devote enough time and energy to the steps of the data analysis process that are important to the broader result. Sometimes, this is because they intentionally ignored these aspects of the data process. More often, though, it’s just that they weren’t aware of the importance of these steps.

Two of the steps that businesses most often ignore or deprioritize are data transformation and data visualization. These two steps are crucial for turning raw collected data into something that can actually be used to drive insight and make decisions.

Much of the time, businesses completely ignore or only do the least amount of work possible on data transformation, and then expect the data visualization stage to completely cover their mistakes. This leads to dashboards with badly built visualizations that end up doing a poor job of conveying business insight.

Data transformation and data visualization are both important elements of a business’s BI strategy. Organizations that rely on data to make their decisions should give both of those steps more attention so that they can build effective dashboards and become more data-driven.

Data transformation

Data transformation might be one of the most commonly ignored steps of the data analysis process, but it’s one of the most important. It’s the step of the process where raw, unsanitized data is turned into useful, actionable data that can be analyzed for insight.

It’s not always a business’s fault that their data transformation gets ignored. Many tools, especially smaller ones with less user input, gloss over the data transformation process or attempt to automate it in some way. These sorts of tools often deprioritize data transformation in favor of more public-facing features like data visualization.

Some businesses are fine with this approach, but in general, it’s better to use a BI tool that allows for a more hands-on data transformation experience. Good data visualization tools shouldn’t come at the cost of other systems, and it’s best to have a tool that can handle both steps of the data process effectively.

Many businesses, especially those that haven’t worked with a BI tool before, don’t understand why data transformation is so important. Data transformation is often pretty invisible, and the difference between an effectively transformed metric and a poorly transformed metric isn’t always clear.

Since poorly transformed metrics are often hard to pick out, it’s hard to know if a metric has been properly transformed or not. To make it more difficult, there’s rarely one ‘correct’ way to transform a given metric. It depends heavily on the quality of the incoming data, the tools at an analyst’s disposal, and the goal of the metric.

Raw, freshly collected data is rarely ready for use in a dashboard or visualization. There are often errors, the data is usually badly formatted, and it generally needs to be compared or connected to other metrics to be of any use. To turn this raw data into something that can be used, data analysts need to transform it.

It varies from tool to tool and from workflow to workflow, but generally, the first step in transforming data is cleansing it. In data cleansing, data is standardized, scanned for errors, and reformatted to fit in with the other data in a business’s data storage structure.

Most data cleansing is done automatically, but there are some aspects that require more input. Software can scan data for typos, errors, and inconsistencies, but it’s generally up to the data analyst to figure out what to do with those errors.

After the data has been cleansed, it’s ready for the real transformation stage. During this stage, data is edited, reconfigured, calculated on, and cut up to make it more useful for building metrics and visualizations off of. There are many sorts of techniques that data analysts can use to transform their data in this way.

What this step actually looks like depends on the tool and the data transformation feature. Most tools allow for raw SQL entry, where users can enter custom SQL operations to transform their data in highly customized ways.

More recently, many tools have introduced visualizers for this process, so that a user without much SQL knowledge can still build simple transformations. These tools use visual abstractions like workflow charts or drag-and-drop cards to hide the SQL from the end user and provide a seamless, low-code experience.

These sorts of tools are more common in self-service BI tools, where the average user is generally expected to have a lower level of data experience. They abstract much of the process but still allow for a high level of control over the data transformation experience.

During this process, data analysts might filter out different data points to make the data more valuable, separate or join rows and columns, use value mapping or string editors to change data values, sort the data set in a new way, or even use a calculated field to find derived values from the data.

Occasionally, users need to combine two or more different data streams to make an effective analytic or metric. Combining data sets is another element of data transformation. During data transformation, analysts can join or append different data sets to make them more effective.

The end goal of data transformation is to get a data stream that can effectively power a metric. The best way to do that will change, given the desired metric and the quality of the underlying data. BI users often need to experiment a bit to find the most powerful transformation for a given data set.

Data visualization

Once a data stream has been turned into a valuable metric, it’s time to turn that metric into a visualization that can be easily understood by the people who need it. Data visualization is another area that many businesses don’t give enough attention to.

Businesses deprioritize data visualization for the opposite reason that they deprioritize data transformation. Unlike with data transformation, where businesses ignore it because of its perceived difficulty, businesses ignore data visualization because they see it as too simple.

There’s a perception that data visualization is the ‘easy’ part of the data process, that it’s so simple to build a graph or chart out of a metric that there’s no need to give it any special thought. While it’s true that often it’s fairly simple to complete the data visualization stage, that doesn’t mean that it should be ignored entirely.

In most BI tools with a data visualization component, it’s fairly easy to visualize a metric in a simple graph or chart. In some tools, the visualization process might even happen automatically, with internal algorithms deciding what sort of visualizations are correct for a specific metric or analytic.

In these sorts of situations, it’s very tempting for BI professionals to take the path of least resistance, to rely on the first visualization that they put together or that their BI tool recommends to them.

This can often be a totally acceptable approach to data visualization. In many cases, using a simple data visualization is more effective than using a complicated one, so this approach can actually boost understanding of the underlying metric. In many other cases, though, this approach can end up harming the overall product.

The issue comes when dashboard builders and analysts use visualizations that don’t effectively communicate the underlying metric, because selecting and using those visualizations was easier than finding a more effective way to express that data. This is one way that the ease of use of modern BI tools can work against data accuracy.

Just like with data transformation, it’s less clear what ‘correct’ data visualization looks like. It depends a lot on things like what sort of metric is being visualized and what sort of visualizations the target audience can easily understand. Similar to data transformation, there’s no right answer, and everything is subjective.

However, the effects of a poorly visualized metric are still visible. Poorly visualized metrics might mislead or confuse viewers, they might include trends in data that aren’t there and gloss over trends that are, and they might end up leading viewers to conclusions that aren’t ideal.

To combat these sorts of problems, dashboard builders and data analysts need to give data visualization the same sort of attention that they would give to other aspects of the data process. Good data visualization doesn’t just happen, and data experts need to put some effort in to make sure that their metrics are effectively visualized.

Prioritizing important data processes

Both data transformation and data visualization are important parts of the data analysis process. However, businesses rarely give these aspects of the data process the sort of attention that they deserve. Many businesses end up with badly transformed data powering badly visualized metrics, which results in much less valuable insight overall.

There’s nothing forcing businesses to ignore these data processes at this level. For sure, many businesses use BI systems that don’t allow for much customization over the data transformation process, and others don’t have much freedom in data visualization, but in any good BI tool, businesses can give each process the sort of attention that they need.

Data transformation is an important element of BI success since it ensures that businesses can turn their raw, unsanitized collected data into something that can actually be used to drive insight. Businesses need to give proper attention to data transformation, or their metrics won’t provide usable information.

Data visualization is just as important. It’s the process of turning a metric from a stream of numbers into a visualization that can easily be understood by casual viewers of a metric or dashboard. Without effective visualizations, users are unable to understand their data and drive insight from it.