What I’m Thinking About after dbt Coalesce 2022

I had a blast attending dbt Coalesce in New Orleans this year. Thanks to the dbt team and the community for putting on a welcoming and thought-provoking event. Over the past couple weeks, I’ve let my hot takes marinate into more subtle, nuanced ones, and I’m ready to share what I’m thinking about following the conference.

CI is one of the cool kids

When Dylan and I first launched Spectacles, we worried that the term “continuous integration” was too-jargony and specific to software engineering, so we described Spectacles as an “automated testing tool” for Looker.

I’m not worried about that anymore. Everyone I talked to at Coalesce was familiar with the concept of CI for their data stack, and most had some kind of CI setup for dbt, Looker, or beyond. With embedded analytics, data apps, and other high-stakes experiences being built on top of an ever-fragmenting data stack, testing your work in a development environment is more important than ever.

Some quick hits on CI:

• Hightouch announced CI checks.
• dbt slim CI was a frequent topic of sessions and conversation.
• I attended an awesome talk on change management.

BI becomes a search & recommendation problem

I’m betting on the idea of the semantic layer, even though I’m not sure which framework or tool will win in the end. So what does a BI tool built for the semantic layer look like?

Will the idea of “headless BI” (i.e. one tool for the semantic layer and a different one for BI) win out? I’m not convinced. I’m betting that next-gen BI tools will have their own charting capabilities, but will surface great APIs and database connectors so other visualization tools can query the semantic layer easily.

From my vantage point, if governance and modeling is “solved” by the semantic layer, the best BI tools will excel at answering questions quickly.

Design and user experience will be more important than ever before. BI tools built for the semantic layer will look more like Metabase and feel more like Google search (see this GPT3 text-to-SQL demo). They’ll incorporate large-language models to enable questions as entry points to queries instead of pivot tables.¹ They’ll use new tech like DuckDB to make these experiences work on your phone, in your browser, and all of it will be fast.

Looker isn’t going anywhere… yet

Don’t get me wrong, I heard a decent amount of grumbling about Looker. Some people I talked to are unhappy campers, complaining about price going up, support and account management deteriorating, and slow product development.

With that said, there’s not yet a compelling alternative to be found. Nothing is as powerful and fully featured for a large data team as Looker. Robust access controls, embedded analytics, impressive API coverage, and years of innovation in LookML don't happen overnight.

We hosted an amazing discussion with Looker users who also use dbt. People shared some really impressive and thoughtful tips and tricks.² Companies are doing astonishingly cool things with Looker, and while the dbt Semantic Layer is exciting, it’s got a ton of catching up to do to even reach parity with LookML (I’m looking at you, joins).

I saw demos of Lightdash (use your existing dbt project), Omni (fast prototyping, then push down to semantic layer), Sigma (spreadsheet on top of a semantic layer), and Transform (crazy tech, semantic layer with built-in CI checks!), each of which blew my mind and made me so excited for the future of BI. However, dbt Labs is easily the best funded player in this space, and the dbt Semantic Layer is not ready to go head-to-head with LookML yet.

If I'm running data analytics in 2023, there’s no way I’m leaving Looker. This year, BI is still Looker’s game to lose.

In case you missed it

In case you missed it, I spoke about testing dbt changes in CI for negative impacts on Looker. For the step-by-step, jump to the 12:23 mark.

We also released a hilarious video made up of fake AI-generated artwork of people throughout history dealing with bugs and errors. I hope it makes you laugh! Here's a Tweet thread that breaks down how we made it.

Notes

1. I don't think LLM-enabled BI is going to begin as a text box with natural language search, but it will probably be a semi-structured query environment with lots of suggesting and autocompletion. For more, read Davis Treybig's article, "The biggest bottleneck for large language model startups is UX."‍

2. One of the most engaging topics from our discussion was this blog post about restructuring your Looker project by Kenny Ning

‍