About Me

My photo
An experienced Data Professional with experience in Data Science and Data Engineering interested in the intersection of Machine Learning and Engineering.

Most Recent Post

2022-04-01

Why does my startup need a Data Team anyway?

Why does my startup need a Data Team anyway?

This is a reasonable question. Any business should have a healthy skepticism to hiring. Trying to hire your way out of a problem is feasible, but can be a really expensive proposition for any company.

Typical questions,

  • Can't the engineers, the accountants and finance people handle data stuff?
  • What benefit do I get from hiring data analysts, data engineers and data scientists?

The engineers, accounts and finance folks can take a business a long way. They can make plots and charts and determine if the business is making money or going bankrupt. They can support quarterly reports to a board of directors.

A Data Team tackles more fast moving, predictive, larger and messier datasets than accounting and finance and free up the engineers to do actual engineering work. Having data specific folks is a specialization of skill and allows focus on a bounded domain.

Let's start with an Exercise

Let's calculate Revenue over last month from a table with columns

  • customer id
  • revenue
  • revenue date time

This is super easy right?

  • Filter by date time to the "last month"
  • Sum the revenue, and you're done!

In practice, the tables are never this simple. They tend to have dozens of columns and revenue data can be spread across another dozen tables. Trying to get a handle on 12 * 12 * number of records for even a small number of records takes some time and effort.

Typically a revenue calculation would involve,

  • refund amount
  • item sold
  • type of item sold
  • different revenue streams that may have their own tables
    • or might even be stored in the same table in different columns
  • etc, etc, more and more columns and tables, on and on and on

Accountants and finance folks can handle these type of problems in Excel with little difficulty. An engineer can run some SQL, hand the data off to accounting and they can load it into Excel. Excel is great, but does have has size limitations. There is only so much memory on a laptop and the business complexity will continue to increase.

More tables, more SQL joins, and more corner cases cause businesses to increase the amount of time required to support basic revenue calculations. At some point, it becomes a half time or full time position for a dedicated person.

Is there a data team?

Accounting and finance may gradually ramp up the work for the engineer to support their efforts. Does your business have an engineer spending more than 50% of their time supporting data pulls? Does it have 2 engineers spending 25% each?

Then your business already has a data team, it just hasn't been verbalized. That's not a great position to be in. Best to have clear communication, goals and expectations on folks.

I would propose that the equivalent of one-half on an engineer's time spent supporting data pulls for accounting and finance is a data team.

What do I mean by "Team"?

In the above example accounting and finance were the core of the data teams, but not all businesses have such a complex accounting or finance group. Some tech startups have more questions on the business operations than they do on their revenue and tax commitments.

  • How many people visited the website today?
  • Where did our new visitors come from?
  • How can we encourage them to stay?

This can lead a business into data analytics. Analysts focus on business questions and business use cases. But their focus is so oriented towards business problems, that they are necessarily the strongest engineers. In order to move faster and have more regularity in their delivered reports they will need support, and thus a business may hire a Data Engineer or Analyst Engineer.

This can cascade into a full data team with analysts, engineers and scientists - What are all these data people doing?

Data teams grow gradually, one person at a time until someone stands up and says "Look Over Here! We have a data team".

Conclusion

Maybe your organization doesn't need a full data team. A single accountant can support a large, complex business, and maybe that's enough.

Maybe you already have a data team, but it just hasn't been said. Folks supporting accounting and finance, folks writing SQL, making plots and charts and just generally wrangling data to provide value to the business.

Being clear with data goals and expectations can help accelerate the value that the data can provide. Daily updated dashboards can help a business understand where it is right now. Historical data analysis can help them see where they have been and forecasting can give a business an idea of where it might be in 6 months.