Data Scientist or Analytics Engineer: How I Made the Decision That Defined My Career
The year was 2017 and I was preparing to hand in my resignation for my job as an analyst at a high growth tech startup. My plan: take a year off and study to become a data scientist. Why give up a job I loved to achieve the dream of becoming a data scientist? At the time the answers were obvious.
Compared to what I saw #datascience influencers saying on twitter, my work just wasn’t exciting enough. Here I was doing algebra, not algorithms. I was setting OKRs, not doing k-means squared. And not only was I not using deep or recurrent neural networks, I wasn’t using any neural networks at all.
As you can see, dear reader, this situation could not be allowed to stand. So I packed my bags, moved to a new city and spent my days applying to data science bootcamps and analyzing that one titanic dataset on Kaggle. But then something funny happened: when my long-awaited acceptance at a data science bootcamp finally came through, I realized that I … actually didn’t want to be a data scientist anymore.
What could possibly have convinced me to give up on my dream?
Around the same time I resolved to become a data scientist, I was lucky enough to be introduced to a new community that unbeknownst to me would cause me to rethink everything I thought I knew about how to be an impactful data practitioner. It turned out that what I wanted wasn’t to become a data scientist, it was to learn how use data in a way that actually helped organizations make better decisions. The only question was how to do this.
The answer came from a funny place — a small slack community for an open source data transformation product, maintained by the team at a boutique Philadelphia based data consultancy known as Fishtown Analytics.
The folks on the that Slack had some borderline heretical beliefs. Instead of focusing on the quasi-mystical arts of data science — they were figuring out how to apply battle tested software engineering best practices to data analysis. They believed that we could teach analysts to rethink their workflows and massively increase their impact by adopting some principles from developers. At the time, when analysts were fiddling in Excel or wading through the depths of a disorganized and siloed data lake, the team at Fishtown Analytics was working with a few daring organizations to put these ideas into practice.
Of course, you know Fishtown Analytics today as dbt Labs, the vc-darling, unicorn and rising star of the data world, but back then that outcome was far from predictable. But to me, and many others, the appeal of this strategy was irresistible.
I think that the decision not to pursue data science and instead double down on (what would become) analytics engineering is going to be the defining moment in my career. Furthermore, I think if I were making that decision again today, it would be even easier than it was back in 2017. Let’s take a look at why I made this decision and why I’d encourage everyone thinking about the next step in their career to consider analytics engineering.
- Analytics engineering solves the problems organizations actually have
There is almost no organization on the planet that could not benefit from having organized, well modeled data that unifies information from disparate sources and allows them to create well organized abstractions and data products. From the three person startups to the fortune 500 companies, from universities to NGOs, almost no one really feels like they have a full view into what is actually happening with the day to day data in their organizations. There are too many inputs coming from too many different places to get a clear picture of what is going on.
Taking disparate, messy data from disparate sources and using it to draw real insights is exactly the problem most organizations are facing and it’s the problem that analytics engineering was designed to solve. When analytics engineering is useful everywhere, then being an analytics engineer means you can go anywhere. I made a bet in 2017 that people who knew how to do these things would be an asset wherever they went and that’s something that is still true today.
And even better — if you happen to already work in a role where you have access to some data — you can get started providing value with analytics engineering at zero cost and likely just a few days’ time investment. All you have to do is pick an initial use case where automated data modeling would save time for you or your coworkers, use an off the shelf ETL tool to load in your data and then get started on your dbt project to automate this use case. Once you’ve done this and proved the value to your organization and more importantly, yourself, you’ll begin seeing opportunities everywhere.
2. Analytics engineering allows you to massively scale yourself
One of the things that convinced me that analytics engineering was for me was when Tristan told me that a key part of the philosophy of dbt (and analytics engineering) is that it allows you to solve hard problems once, then gain benefits from that solution indefinitely. Historically many data or data adjacent jobs have involved a huge amount of repetitive, manual drudgery. Whether it’s hand crafting the same monthly spend report in Google Sheets or going into your marketing automation system and downloading five different csv files and Vlookuping them together, these tasks quickly begin to eat up your days and drain your soul.
Spending your days working on manual tasks has two drawbacks. First — it is incredibly boring. Second, you pretty quickly hit a ceiling on the amount of reports and projects you can manage as these repeating tasks pile up and you simply don’t have time to do anything else.
By following analytics engineering principles and best practices and by learning to solve hard problems once, you can reach levels of output and productivity that would be literally impossible beforehand. Within a few months of applying them to my analyst job, I had essentially gotten rid of 80% of my old manual workload — time that was freed up for me to work on new and more interesting projects that ended up becoming incredibly valuable for the org and my career.
Learning to master the tools of the analytics engineering trade essentially allows you to turn yourself into an entire data department of one, constantly solving new hard problems instead of the same ones over and over and over again.
3. You don’t need a phd to be a world class analytics engineer — you just need to be purple
One of the most exciting aspects of analytics engineering is that the barriers to entry are much lower than for more traditional roles like data scientists or data engineers. There is a recognition among the community that even though a certain degree of technical proficiency is necessary to become a great analytics engineer, technical skills are not the most important aspect of being an analytics engineer.
Once a threshold of technical proficiency has been attained, curiosity, empathy and laser sharp communication are the differentiators that really make you stick out.
And you don’t need a phd, a master’s degree or even any formal training at all to demonstrate these. That means that analytics engineering is a field ripe for entrants with nontraditional skill sets and backgrounds. Not only is it possible to enter the field this way, many (if not most) of the most impressive analytics engineers I know have done so. Most of us started the same way, learning sql then layering dbt on top. Once you’ve mastered the core concepts in those, you know a good amount of the concepts you’d need to use for an entry level Analytics Engineering role.
And that’s great news, because at a time when it is harder than ever to land a data science job, the number of listings for analytics engineers has been increasing exponentially.
Conclusion
This is a really exciting time to get involved with analytics engineering. Compared to just a few years ago, we now have a conference, a slack community at 16,000+ members strong (including a thriving jobs board) and a burgeoning ecosystem of analytics engineering tools. But these are also still, as Oculus Chief Scientist Michael Abrash likes to say “the good old days” — the very beginnings of the time when we can all collectively figure out what exactly Analytics Engineering is as a discipline.
Ten, or even five years from now — it will be obvious to everyone that Analytics Engineering was always going to look like XYZ. But we don’t live then and as of right now, XYZ is still being collectively written by all of us. If you’re the type of person that wants to help decide what that future will be, then Analytics Engineering is for you. Come and join us.
Jason Ganz is the Manager of Developer Experience at dbt Labs.
