The Cost of Untested LookML

Looker is a powerful tool that helps data teams answer complex questions quickly. However, if LookML is not thoroughly tested, it can lead to dashboard outages and frustrated users.

These outages can have a significant impact on the trust that users have in the data team and their ability to make timely and accurate decisions. The usual solution—manually testing LookML—is tedious and time-consuming, taking your team's time away from higher-impact analytics work.

My co-founder, Dylan Baker, and I experienced this problem repeatedly while developing LookML and administering Looker for over a dozen companies over the past decade. We've seen first-hand how untested LookML handicaps analysts, degrades code quality and performance, and reduces trust in the data team.

Curious to quantify how untested LookML might be affecting your company? To make the problem more tangible, we'll imagine a scenario that plays out nearly every day at companies using Looker. Then, after this short example, we'll explain how to calculate the cost of untested LookML at your company.

James is a product director at Shamazon, a large e-commerce retailer that sells replica luxury goods. He's looking for some info on funnel and conversion rates for the product line of watches that he manages.

He goes to Looker, where he has a dashboard bookmarked with stats that he pulls each month for his meeting with the CMO. Except he's unable to pull the numbers he needs because the dashboard has a query error that prevents it from running.

He messages the data analyst who helped him set up the dashboard, "Any idea what's happening here? I need to get these numbers ASAP."

Let's pause for a moment and think about the costs incurred to James' company while this problem gets fixed.

The cost to the user

James's analysis is delayed while he waits for the data team to fix the dashboard. Maybe he's unable to get the results in time for his meeting, postponing a critical decision.

Cost: James' time + cost of delay of his analysis.

James loses trust in the data team and in Looker. "Maybe I'll start pulling these numbers directly from Shopify," he thinks. In the worst case, James decides to do his analysis outside of the governed data model in Looker, possibly resulting in mistakes or poorly informed decisions.

Cost: Probability of James doing the analysis incorrectly outside of Looker x impact of bad decisions resulting from those mistakes.

The cost to the data team

An analyst on the data team has to break away from their planned work to put out this fire and fix the dashboard.

Cost: The data analyst's time to fix the issue + the cost of delay of their planned work.

The next time the analyst makes a change to LookML, they spend time checking each of the company's important dashboards to make sure their change didn't break anything.

Cost: The data analyst's time spent manually and exhaustively testing their work.

The cost to the customer

In this case, the dashboard user is a company employee (James), but for Looker embed (Powered by Looker) customers, the user could be a company customer. In that scenario, the cost is even higher as it results in a negative product experience which could directly impact revenue.

Cost: Potential loss of customers and revenue.

Calculating the cost of untested LookML

Now that we've explored the cost of untested LookML, let's look at how to calculate it for your company.

Untested LookML costs your business in two ways:

• Wasted time spent manually testing for errors and triaging issues
• The opportunity cost of outages (what could have been accomplished by users if they had immediate access to what they need)

Looker developers spend about 15% of their time in Looker testing and debugging errors. Feel free to tweak that number if you believe it to be higher or lower for your team. You can determine the cost to your business with this equation:

Number of Looker developer users × Average Looker developer's annual salary × Estimated % of developer's time spent in Looker × 0.15

For example, for a Looker instance with 50 developers, with an average annual salary of $110k, spending 60% of their time in Looker, you would estimate a cost of 50 × $110,000 / year × 0.6 × 0.15 = $495,000 / year, or roughly the cost of 4 additional developers.

It's harder to calculate the opportunity cost of these errors. Imagine the same company has 150 Looker users, who each spend an average of 1 hour each month waiting for the data team to fix outages. Assuming a 40-hour workweek, that's about 0.6% of their time. If we assume a slightly lower annual average salary of $80,000, the cost is 150 × $80,000 / year × 0.006 = $72,000 / year.

All told, that's a cost of $567,000 to the business in wasted time. You could hire five more analysts with all that money!

At Spectacles, we want to reduce wasted time and make your Looker developers more productive and efficient. So we've developed solutions that will drastically reduce the time your data team spends testing and debugging LookML and handling unexpected outages. Your developers and your users will thank you!

"Spectacles helps me sleep at night, knowing that I can safely merge changes to my project without breaking any vital reports in Looker."

Erica Louie, Head of Data, dbt Labs

“We’re loving the confidence that Spectacles gives us as we rapidly build out our LookML. It’s an integral part of our stack and workflow.”

Rocky Martin, Analytics Engineer, Vendr