The 5 Steps of the Data Visualization Lifecycle

Data visualization is increasingly prevalent, especially in organisations that are driven by data or have ambitions to be. Data visualization is essential to how contemporary businesses make an impression. Every tool and workflow includes it. Not only for data engineers, scientists, and analysts, but also for those without "data" in their title, it's a crucial component of the job. Presentations of products, spontaneous chat on Slack or Teams, leadership reports to shareholders, and even marketing materials all use it.

The best way to approach data visualization is to separate it from roles and tools and instead concentrate on the context in which it is used. Each of these distinct phases, from the exploratory data analysis of raw data to the validation of hypotheses and explaining patterns in the data to the productization of the charts created into regular reports and other data resources, requires a specialised set of data visualization capabilities.

Your data visualization features should take into account how consumers utilise data today, not how they did it a decade ago. Regardless of their position or the issue they're trying to solve, the expectations of data professionals and consumers has increased and converged.

The days when businesses struggled to find data are long gone; today, all businesses struggle to discover the correct data and deliver the relevant summaries of it to the appropriate audiences.

Let's examine each phase to observe how data visualization is utilised and supported in the present:

Step 1: Exploratory Data Analysis (EDA)

When data visualization is used to understand the form and patterns in data rather than to explain those patterns, this process is known as exploratory data analysis (EDA). Although EDA is most frequently mentioned in relation to data science (with tools like ggplot2 and vega-lite suited for the technique), the challenge of data access serves as the greatest illustration of the concept.

Data engineers are continually faced with using data visualization to demonstrate the form of data sources, the lineage of the data, and how it may be connected with other data in order to support this. A data engineer employed data visualization to create and assess a dataset before it was even a glint in the eye of an analyst or data scientist. Many of these visual depictions of the data source are removed once the dataset or pipeline is complete, but some may continue to exist to offer continual reports on the state of the data sources.

A few rows from a table are almost always examined as the initial step in any data task, not because that is the best way to visualise the data but rather because tables are compatible with almost all datasets. Stakeholders frequently opt for a tabular view since it is quick and easy to use when they only need access to data and an overview of it.

EDA is particularly prevalent in the data science field, where it initially begins in a manner akin to that which has already been described but quickly transitions into more focused methods that fall into the subsequent stage of the data visualization lifecycle.

Step 2: Hypotheses Generation & Validation

Using data visualization to develop and test hypotheses is the element of data visualization that is most job-oriented. This is similar to EDA but is more focused because it has gone beyond simple investigation and has made explicit claims about the data.

Tools like ggplot2 and vega, which feature extensive capabilities like faceting and the flexibility to work with practically every sort of data, are used in data science workflows for hypothesis creation and validation.

Other stages of the data visualization lifecycle lack the ability to demonstrate statistical significance and uncertainty, but these technologies often do. To enable hypothesis validation for non-data scientists, statistical tests, especially A/B testing, may make use of more customised interfaces and sophisticated table representations of statistical summaries.

Machine learning is the other significant field where data visualization plays a significant role in hypothesis formulation. In support of machine learning workflows, data visualization might take on a very different shape where the objective is to optimise a specific number (for example, a component of a confusion matrix) for the validation of your hypothesis and then visualise random samples to try and confirm a lack of bias.

Step 3: Explanatory Analysis

A hypothesis must be explained to a group of individuals because its validity alone is insufficient. Even when there aren't competing techniques that are equally viable, organisations may not have unlimited resources to pursue each one. Data visualization must be compelling to those who will be making decisions based on the data being visualised in addition to being clear to the person who created it. Practitioners often have a serious blind spot in this area and are shocked when the charts they used for analysis don't work as well for presenting.

The next step is to clarify it for an audience that is unfamiliar with the dataset and method used by the original developer, even if the hypothesis is simply "This thing is important." Both traditional BI tools and data visualization libraries, which offer the option to style and embellish austere and cluttered charts developed during earlier processes, are used to demonstrate how this is done.

Editing, context, and clarity are three communication elements that underpin successful informative visuals. The exploratory colour schemes, which are designed to display as many distinct values as possible, are changed to more deliberate colour schemes that highlight important themes in the data being examined. The labels on the chart's elements, such as the axes, are less conspicuous and more carefully formatted. The chart receives a title and other text to place the viewer, as recommended by best practises outlined in numerous data visualization guides. Explanatory graphics are developed with an audience other than the person who generated the chart in mind. This is further distinguished by annotations and contextual charts.

Step 4: Into Production

A chart must be circulated and read by its audience, thus making it readable isn't the last thing to do with it. Unless they deal with dashboards, in which case they often believe that adding charts to a dashboard constitutes the only kind of productization, the majority of data visualization guidelines overlook this stage. However, charts also reach audiences in other ways, for as through automated emails, presentations, or memoranda.

Charts are improved for collaboration (such as permitting commentary), easy sharing, easy interaction, and automatically updated through productization (or regularly published in the form of an email report).

Productization could therefore be just as difficult and costly as developing a wholly unique analytical programme, as data visualization engineers at businesses like Apple and Netflix do. Or it may be as easy as sharing a screenshot of a chart during a meeting by embedding it in a paper. The sharing of dashboards created with modern BI applications is made easier by features like distributing them as email reports. Dashboarding libraries like Dash and Streamlit, which allow users to quickly deploy dashboards right out of the EDA and hypothesis generation mode, are located somewhere between custom apps and BI systems.

The most contentious of all of these practises would be including a chart graphic in a paper. Could productization be as easy as pasting the chart into Google Docs, Notion, Coda, Quip, or Confluence? The key requirements of productization are frequently the need for simple sharing and commenting features, and those goals are often met by using static screenshots in online documents. Is it ideal? Not at all. The chart can no longer be dynamically updated, and the individual taking the screenshot runs the risk of unintentionally cropping out crucial information. The benefits of being able to share and comment on the chart, however, are clearly worth those costs, as evidenced by the prevalence of this strategy.

Step 5: Strategic Approach

Productization may also seem like the last step but it’s not. Charts contribute (both adversely and constructively) to knowledge sharing, best practises, and rules for using data beyond their immediate impact on display. Charts are an organization's lifeblood. Only by evaluating how they have utilised data visualization will that organisation be able to enhance how it employs data visualization.

Charts influence a company's strategic direction even when no active evaluation is taking place. Charts highlight and condense metrics. The indicators we present, particularly those that make it from exploration to productization, are the result of significant investment. They affect choices, but they also affect subsequent metrics. Data visualization is a crucial component of Metric Design because of this.

Similar to this, visualization is required for both the data and its transformation. Data lineage involves not only the ETL process providing the data but also the procedures necessary to make that data semantically meaningful enough for an organisation to utilise it for decision-making.

Finally, every chart that a company creates is a chart that its employees see. That may seem like a simple concept, but charts show data in a way that can either increase or decrease data literacy. If all of your graphs are just bar or line graphs, then all of your metrics will only be those that can be used to populate those graphs, and all of your decisions will only be those that can be based on those metrics. However, if you have maps, flows, hierarchical data, topological data, charts that depict uncertainty, and other data kinds, your organisation will be able to base choices on that data. Consequently, even following the successful deployment of a chart, the data literacy of your organisation is still being impacted by it.