Data-Driven Design: Dare to Wield the Sword of Data – Part II
In my previous article, I talked about how web design can benefit from data, and now I’d like to recommend some ways in which data can be better fused with the creative process.
Although recent studies have debunked the myth of left- and right-brained dominance, the analogy still resonates with many people. Even if you feel as though you’re hardwired one way, in order to be successful you increasingly need to use both sides of your brain. However, for many quants and creatives it can be intimidating to venture outside of their normal comfort zone. I’ve found that having a process or guide can be helpful if you’re entering unfamiliar territory. Before I explore how data can play a more prominent role, let’s evaluate the standard web design process.
As I prepared to write this blog post, I researched a number of web design processes from different interactive agencies. Most of the web design processes could be summarized as having four main steps:
1. Plan (Strategy/Vision)
2. Design
3. Develop
4. Deploy (Launch/Deliver)
Having worked for a leading interactive agency more than ten years ago (Blast Radius), the overall process hasn’t changed much in that time. Conversely, web analytics (or digital analytics as we now like to call it) has evolved significantly from the basic, traffic-focused WebTrends and Urchin reports that I used as a marketer back in 1999-2001. Today advanced digital analytics tools from Adobe, Google, IBM, and others provide deeper, more actionable insights into digital performance.
Despite this change in the availability and business value of online data, data was rarely cited in the web design processes I found. Some of the processes mentioned upfront research consisting of user testing or competitive research in the Plan stage, but there was no talk of focusing on key online metrics or evaluating past performance with web analytics. In addition, the last step of the web design process (Deploy) never mentioned A/B and multivariate testing or even evaluating the results of the new site for possible enhancements.
If web design is one of the most measurable forms of design, why aren’t more web designers using data in their design approach? There are probably lots of excuses—not enough time, no access to web analytics or testing tools, clients aren’t asking for it—but I believe it’s long overdue for more data to be used in the web design process. In fact, Rob Randolph, Marketing Director at Cuker Interactive, felt that digital agencies where design is not increasingly informed by data are doomed to become dinosaurs, and he stated “by combining truly inspired creativity with business intelligence, you stand to win more often than you lose.”
Today, web designers have a wide array of analytics and optimization tools at their disposal. You have enterprise solutions from Adobe (SiteCatalyst, Discover, Insight, Test&Target), the ubiquitous Google Analytics, open-source options (Piwik, Open Web Analytics), and a ton of other unique point solutions (Crazy Egg, Clicktale, Mixpanel, Tealeaf, Optimizely). Let’s see how you can put them to better use in your web design efforts.
Rethinking the Web Design Process
If we deconstruct the standard web design process, we can see how data can be injected into the process to drive more success. I’ve identified six different ways in which data can and should be used within the web design process. The gray areas represent basic steps that need to be a part of every data-driven design. The blue areas are equally important steps that will scale in terms of the level of effort with the size, scope, and cost of the project. For example, for a simple display ad project might not need extensive insight gathering or A/B testing; whereas a major site redesign should include all of the data-related steps.
A. Insight Gathering
In most cases during the planning stage, you have access to all kinds of useful information collected in your web analytics tools. Rather than blindly forming a theory of what’s happening on the site based on different opinions, you can evaluate actual visitor behaviors and understand what’s really happening. Too often digital agencies tear down a website and re-design it without clearly understanding from the data what was and wasn’t working. Without data you could actually waste time trying to fix things that aren’t even broken (before you started).
Data can also help you better understand who you’re designing for. Rather than building a persona based on different assumptions, you can use the data to inform your personas in terms of what devices or browsers they use, where they are located, how they find your site, what content they consume, etc. Insights from the data can shed valuable light on your design project, and ultimately help refine your creative approach.
B. Clarify Goals & Metrics
When the desired outcome or goals of a design project are not clear, it is difficult to design for success. If you know what the business goals are, you can focus your design efforts accordingly. It’s better to demand clarity in the planning stage as it will help your design to hit the mark. Business objectives by themselves can be open to interpretation so tying them to specific metrics and targets can help to solidify what needs to be accomplished. The additional clarity and commitment that comes from agreeing upon the success metrics upfront will give your design a greater chance of success in the end.
Don’t fall into the trap of assuming what the key metrics will be without confirming what the business goals are. In many cases, if you don’t properly identify the right metrics upfront, it will be impossible to back into the numbers after the fact. What wasn’t tracked or measured is lost forever, which is challenging if you’re trying to understand how your design performed.
C. Deploy Analytics Tags
In order to measure the success of any web project, you need to have the right tracking in place. It sounds obvious but too often this step is overlooked, rushed, or trivialized, which results in missing or poor data. If you want to get meaningful insights out of your web analytics reports, it’s in your best interest to make sure someone has put some thought into the data collection and validated that it has been done correctly.
As an digital analyst, it can be painful to watch an expensive campaign or website launch when you know there’s inadequate or no data flowing in from it. If you’re not as geeky about data as me, it’s like not having a camera available for your kid’s first steps or when you bump into your favorite movie star or athlete. It represents an opportunity missed that can never be reclaimed.
In addition, if you’re just placing a basic tag with no customization, you’re passing up lots of valuable insights. Investing 20% more time in analytics tagging (we’re shooting for “ample” not “excessive” tagging) could easily translate into 80% more value out of your digital data. While some data is often better than no data (unless it’s bad data), I’ve found the data that has been tailored to your specific business needs will be far more relevant and useful than many of the default metrics and reports in your analytics tool.
Ideally, this step of setting up and validating the analytics tagging should be baked into your web development process so that there’s ample time and attention to do it right the first time. In addition, your web infrastructure should streamline, not complicate, this step. If you’re constantly battling an outdated content management system, your company will need to address this problem. Your web infrastructure can’t be an excuse for inadequate tagging and subpar insights.
D. A/B or Multivariate Testing
While web analytics can provide you with some useful insights for your design, nothing beats causal data where testing reveals what performs and what doesn’t. Designers who embrace optimization are often shocked at what creative works or doesn’t work when it is tested on actual users. Through testing you can better understand what influences the desired outcome, enabling you to hone your design before it is pushed to all of your online visitors. You also have the opportunity to test multiple creative variations rather than settling for just one design idea or approach. You might be disappointed when users don’t respond to a particular design, but you’ll learn more about their preferences as you continue to explore who they are and what they want.
At Adobe, the designers participate in monthly sessions where they review what performed and what didn’t in their testing campaigns. The web design team looks forward to those meetings, and the attendance has grown to more than 30 people. Senior Art Director, Ben Child, commented that it’s always interesting and surprising to see what performs for different audiences. As an example, they found that digital marketers respond to very different creative than what is used for design professionals.
E. Post-Launch Results
Rather than rolling onto the next design project without looking back, it’s in your best interest to understand if your design worked or not. In the fast-paced world of web design, not enough time is reserved for reflection and examination. If you had clear business objectives and captured the right metrics, it should be apparent how your design performed. The post-launch results form a baseline and provide valuable feedback on your design. If you check your ego and have an open mind, you’ll learn a lot from the insights hidden within your analytics data. They not only give you an opportunity to quickly respond to user experience issues, but the insights may inspire new directions in the future.
I’d recommend monitoring the key metrics after the launch and having a post-mortem review after the dust has settled and visitors are accustomed to the new experience (2-4 weeks). If you don’t have an analyst who can help with the analysis, don’t hesitate to learn how to use the web analytics tools so you can obtain the insights you need. In most cases, you don’t need an advanced degree in statistics to use these tools, just curiosity and a desire to learn.
F. Ongoing Optimization
They say design is never done. Even when a good design improves an online process or provides a better user experience, there is always room for iteration and improvement. In fact, a series of small improvements may eventually translate into a significant enhancement for your business over time.
One of the criticisms of testing or optimization is that you can fall into the trap of optimizing to a local maximum, focusing on only small enhancements and never making any daring design decisions. While it might be easy to blame data for standing in the way of bold design ideas, it is misplaced. The internal sacred cows, conventions, accepted practices, and heuristics at your company are actually what impede design innovation. These implied constraints put the data and testing in a box that drives them toward the only place they can go—a local maximum.
Whenever you challenge a perceived internal constraint or rule of thumb (e.g., “our home page always needs a rotating hero banner”, “our navigation menu has to be horizontal”, “only certain products can be featured on landing pages”), you remove the barriers to both your creativity and what can be improved. If the business goal is to increase a specific metric, then everything should be on the table (no untouchables) or you must accept a local maximum. If you believe in continual improvement of your design, then data is a valuable contributor to ensuring your design is both daring and successful.
If it doesn’t sell, it isn’t creative.
David Ogilvy, Advertising Executive
Beautiful design and elegant code are wonderful. Good job, you get a gold star—maybe even a Webby or CLIO award. However, a more effective user experience that drives significantly higher online sales (leads, subscriptions, engagement, etc.) is really what makes a design shine. I agree with Ogilvy that creative that drives positive results is what really matters. You’re only going to know if your design worked with data, and optimization can help your design to reach its full potential. Don’t fear the sword of data, master its insights to your advantage.
Data-Driven Design: Dare to Wield the Sword of Data – Part I
At times, creative and data are seen as being at odds. Some designers view data as a potential barrier to their creativity and feel their designs shouldn’t be judged solely by data. Three years ago a flashpoint in the “design vs. data” debate happened when the top visual designer at Google, Douglas Bowman, left the company after being frustrated with its oppressive, data-driven approach. On his personal blog, he shared his perception that data eventually had become “a crutch for every decision, paralyzing the company and preventing it from making any daring design decisions . . . I won’t miss a design philosophy that lives or dies strictly by the sword of data.”
It was interesting to see the reaction in the design community to the whole data-driven design debate. As designers Lukas Mathis and Tom Chi noted, a key difference between art and design is constraints. Whereas artists have relative freedom to do what they want, designers must create a solution that works within certain parameters. An unknown source stated, “Design is the art of gradually applying constraints until only one solution remains.” There’s a false notion that data kills creativity. I believe creativity actually flourishes with restrictions. One of these key constraints is the performance of your design. For example, does your design help more people to accomplish their online tasks? Does it drive more online conversions for your business? If the effectiveness of your design matters, then measurement and data are essential.
Design is not just what it looks like and feels like. Design is how it works.
Steve Jobs, Co-founder of Apple
Author and designer Luke Stevens blamed Google’s broken decision-making culture for Bowman’s challenges and disagreed with Bowman’s position. He felt web design should live or die by the sword of data, and that designers shouldn’t view the data as some abstract thing but “a person like you or me trying to do something, and succeeding or failing.” He went on to say, “For perhaps the first time in history we can accurately measure all interactions with a piece of design. If it can be measured, it can be improved. And each of those improvements represents helping someone do something a little more successfully.” If more web designers fully appreciated what online data represented (i.e., attributes, interests, and behaviors of real people), then I reckon more would to seek to wield the sword of data in their craft.
Different Degrees of Data Adoption
Some people might take the position that you just need a healthy balance between using your intuition/gut and focusing on data. It’s wrong to think of intuition and data usage as being on opposite ends of some continuum, and it’s all about finding equilibrium between the two points. It’s more about understanding how attuned your intuition is to what really works. Sometimes you’re not going to have data to lean on. In those cases you’ll need trust your experience and instincts to put the best foot forward and then re-calibrate as needed once you have data. While I do think there are different levels of data adoption, it’s definitely not about finding the middle ground between intuition and data.
I’ve found that the term “data-driven” is a loaded term for some people, and I don’t want semantics to stand in the way of changing opinions and behaviors in this area. By data-driven I mean data is being actively consulted for information and insights that can shape and inform a design. Some people prefer using “data-informed,” but I believe it really represents a weaker, more passive stance on data. On the surface, data-informed and data-driven individuals act the same way when the data supports their intuition. However, what happens when the data goes against your intuition? That’s where the two approaches diverge.
If you’re data-informed, you’ll always side with your intuition over whatever the data tells you. You’ll only use the data if it’s useful in advancing your position or supporting your decision. If it strays from your intuition, then it is no longer useful and discarded. However, if you’re data-driven, you’re more likely to trust the data even when your intuition disagrees. You view these situations as an opportunity to calibrate your intuition and learn. Some people misinterpret data-driven as “data-controlled” where humans lose all control of decision making. Intuition and data should operate as checks and balances. I also believe you can be data-driven and still decide to follow your intuition from time to time. However, in these intuition-over-data situations, you’re fully aware of the potential consequences and prepared to closely monitor the results of your design decision and re-calibrate as needed.
There’s another key mindset difference between data-informed and data-driven. If you’re data-informed then you view data as nice to know; whereas, if you’re data-driven, you view data as need to know—it’s essential to your role. It’s a subtle but important difference. In addition, being data-driven requires effort. A great deal of discipline and diligence goes into seeking out data throughout the design process. It really bothers you when you don’t have the data you need. If you don’t like the term “data-driven,” you might prefer Joshua Porter’s concept of metrics-driven design. Regardless of your semantic preference, consider how using data in your web design can have a positive effect on you, your organization/client, and its online visitors.
Qualitative and Quantitative Data
If you’re interested in data-driven web design, there are two main types of data at your disposal: qualitative and quantitative data. Qualitative data focuses on people’s expressions of their opinions, feelings, and behaviors. When analyzing qualitative data, you’re interpreting the meaning of the words that were shared. Common forms of qualitative data include usability testing, focus groups, and open-ended feedback from online surveys. In contrast, quantitative data focuses on numbers or statistics that are collected through web analytics, A/B testing, or customer satisfaction scores. Many web designers are more comfortable with qualitative data than they are with quantitative data. This needs to change, especially if you’re using neither type.
Each form of data has inherent advantages and disadvantages. Qualitative data can answer the elusive “why” and provide valuable context. Quantitative data helps to clarify who, what, when, where, and how. On the negative side, qualitative data is highly subjective and limited in scope to small samples (e.g., three people in the focus group said they hate the blue color so we can’t do it in blue). Quantitative data lacks contextual detail (e.g., I know most visitors are abandoning on a specific step in the process, but I don’t know why). Whenever possible it’s beneficial to leverage both data types as they can complement each other. For digital initiatives, which are highly measurable, you can access rich levels of quantitative data that can shape and inform your designs. If you’re only using qualitative data, you’re missing out on the vast majority of the available insights that can be gleaned from online data.
Here are some examples of the types of quantitative data and reports you can leverage in your web design:
Referral sources: Where are your visitors coming from? How are they finding the website?
- Referral domain or URL
- Marketing campaign or channel
- Search keywords
Experience parameters: How are they experiencing the site?
- Device type
- Browser type
- Screen resolution
Content popularity: What content are they interested in? What are they looking for?
- Articles
- Videos
- Downloads
- Site sections
- Internal search terms
Engagement: How compelling or engaging is your site?
- New / Return visitors
- Return frequency
- Landing page bounce rates
- Time spent on site or time spent on specific pages
- Page views per visit
Task completion: What are they able to accomplish? Where are they running into problems?
- Goal / Conversion funnels
- Key navigation flows / Page pathing
- Page heatmaps
- Attrition or exit points
- Page load performance
- Micro-conversions (downloads, trials, demo views, product views, etc.)
- Macro-conversions (orders, leads, subscriptions, applications, etc.)
Overall success: How successful is the site, app, or campaign?
- Revenue
- Macro-conversions (see above)
- Conversion rates on key macro-conversions
- Marketing costs, ad spend, etc.
Customer satisfaction: How happy are your customers with your company? How satisfied are they with your website?
- Net promoter scores
- Customer satisfaction scores
Optimization: What influences or contributes to online success?
- A/B testing
- Multivariate testing
With the help of this data, you have an enriched understanding of your target audience and how they’re interacting with your designs.
Data is a two-edged sword
You can be a victim of the sword of data, or you can wield it to your advantage. Data is a two-edged sword that can both challenge and empower you as a designer.
One side of the data sword challenges designers. Data sets a higher standard for web design—one that is objective and not open to interpretation. It holds web designers more accountable for the effectiveness of their designs, which may make some uncomfortable. Data will present challenges that will require you to solve unexpected problems and iterate on your designs. As organizations become increasingly data-driven and place more emphasis on success metrics, you can choose to ignore the data—but no longer without consequences. Just as companies are evolving in their use of data, web designers must adapt and embrace data to increase the efficacy of their designs.
The other edge of the data sword empowers designers. With greater insight into your visitors and their unique behaviors, you are better equipped to design for their needs. If you are designing without data or fail to understand the results of your designs, you’re also passing up the opportunity to learn and improve. Data will validate or invalidate your assumptions, and it can refine and sharpen your design approach over time. In addition, the metrics can be a common language between web designers and business groups. Armed with data, designers can better defend their creative decisions and elevate the focus away from dueling opinions of the business users, executives, and the creative team. It can speed up decision making and gain buy-in from key stakeholders by adding a sense of urgency (e.g., this problem is costing us $$$ each day we wait). Data can also generate more visibility for the impact your design work is having on your organization.
There’s a difference between holding a sword and wielding it. You can grasp the sword of data awkwardly and not use it effectively. Alternatively, you can wield it with skill and ease in your web design. Just like with real swords, you’re not going to master the sword of data without practice so don’t become frustrated in the beginning. With time you can learn to master its power in your design efforts.
In my next post, I’ll share how data can be integrated into web design processes to make it more data-driven.
31 Essential Quotes on Analytics and Data
Who doesn’t like a good quote? We come across interesting and useful quotes all of the time in literature, news media, entertainment, and so on. A potent, succinct quote that underlines a key point or supports an important truth can be like gold. Often the value of a quote is reinforced by who stated it such as an industry expert, a well-respected figure, or a person of notoriety, but sometimes the quote may be so good that it can stand on its own even if it came from a lesser known or even unknown source.
If you’ve read my web analytics book, you’ll know that I used quotes throughout the book to emphasize several points. I thought I’d share some of my favorite quotes related to analytics and data—some that made it into the book and others that didn’t (now I’m debating why I wasn’t able to squeeze them in). If you’re evangelizing web analytics or trying to nurture a data-driven mindset at your company, I hope you’ll find these 31 quotes inspiring and helpful.
Why Analytics?
At the end of the day, this quote by the late management guru, Peter Drucker, represents why most companies invest in analytics. If you’re not measuring your digital initiatives, how can you manage or optimize them? Now just because you measure something doesn’t guarantee that you will actually manage it, however, measurement is the starting point.
The price of light is less than the cost of darkness.
Arthur C. Nielsen, Market Researcher & Founder of ACNielsen
Another way to look at your analytics investment is to evaluate what it would be like to NOT have information on your digital business. I like how Arthur C. Nielsen, the founder of the global market research company, ACNielsen, framed it. Sure you need to spend some money on analytics (technology and people), but it’s going to be less than the price you’ll pay if you don’t know how your business is performing. Consider the missed revenue opportunities and potential cost savings you would be giving up.
Without big data analytics, companies are blind and deaf, wandering out onto the Web like deer on a freeway.
Geoffrey Moore, Author of Crossing the Chasm & Inside the Tornado
Without analytics, your company is vulnerable. However, just owning analytics tools isn’t enough. Extracting value from your analytics investment requires sufficient staff, well-defined processes, a clear business strategy, and leadership support.
In the context of analytics, I’d like to modify this quote a little by substituting “understand” for “remember.” Analytics gives you valuable insights into what is and isn’t working. If your company doesn’t understand (and remember) what mistakes it has made with its websites, campaigns, or mobile apps, it will keep on making those same errors.
War is ninety percent information.
Napoleon Bonaparte, French Military and Political Leader
The famous French general didn’t even live the information age, and yet he attributed most of his military success to having the right information. When you’re battling for a competitive advantage in business, analytics data can be equally important to your success.
Uninformed Opinions and Theories
A point of view can be a dangerous luxury when substituted for insight and understanding.
Marshall McLuhan, Canadian Communications Professor
Rather than blindly following the most popular or loudest opinion, you should use data to question, inform, and shape your perspective. While you don’t always have to agree with the data, your understanding will be increased and enhanced by it.
This quote, which is often attributed to manufacturing guru and statistician, W. Edwards Deming, encapsulates the data-driven mindset. Unless you’re an omniscient, omnipotent being (or Chuck Norris), you’d better have data to back up your idea or approach.
I never guess. It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.
Sir Arthur Conan Doyle, Author of Sherlock Holmes stories
Whether you approach your data seeking enlightenment or to prove a point, you’ll find what you’re looking for. Although a common practice, twisting or massaging data to support a theory will only lead to problems down the road.
He uses statistics as a drunken man uses lamp posts – for support rather than for illumination.
Andrew Lang, Scottish Writer
Some managers are only interested in the numbers when they support their decisions. Rather than dismissing bad results or covering up mistakes, more managers should embrace these trying insights for the value they can provide in helping to refine their understanding, intuition, and overall effectiveness.
Just like many other things in life, data can be ignored. It’s true we might not always like the results we see in our analytics tool, but ignoring the data isn’t going to make things any better either. Do something about the numbers, don’t ignore them.
Data Matters
Data! Data! Data! I can’t make bricks without clay!
Sir Arthur Conan Doyle
Sir Conan Doyle’s famous fictional detective, Sherlock Holmes, couldn’t form any theories or draw any conclusions until he had sufficient data. Data is the basic building block of everything we do in analytics: the reports we build, the analyses we perform, the decisions we influence, and the optimizations we derive.
Not everything that can be counted counts, and not everything that counts can be counted.
Albert Einstein, Physicist
While data is important, the right data is essential. It’s becoming easier to feel overwhelmed by the increasing amounts of data being collected. Understanding what’s important to the business (you need clear business goals) helps analysts to evaluate what data counts or should be counted. Albert Einstein was on to something when he said we can’t always track what we want to count, but that doesn’t stop us from constantly exploring new ways to get the data we need (just remember to respect privacy policies and laws).
The Process of Analysis
In analytics, success is dependent upon asking the right questions—both of the business stakeholders and of the data in your analysis approach. Framing a business question properly is critical because if you don’t, you could go astray in your analysis or return with the wrong insights.
A person who is gifted sees the essential point and leaves the rest as surplus.
Thomas Carlyle, Scottish Writer
A successful analyst is able to remove the noise when analyzing data and isolate what matters to his or her organization. With most companies collecting large amounts of data, you need to be both talented and disciplined to pinpoint key insights that can yield value.
Sometimes you can get lucky and discover low-hanging fruit with minimal effort. However, often you’ll need to go deeper than the surface-level information to uncover the valuable insights hidden within your data. For example, nothing may standout when you first look at your conversion funnel reports, but when yo