Generalists Are The Real Data Science Unicorns

Rarer Than Rocking Horse Droppings.

I started chasing unicorns early in life. 1983. Four years old. Iveagh Cinema. Watching the first movie I ever saw on the big screen – The Last Unicorn. The beautiful old-style movie theatre I saw it in has since been knocked down to build houses. The price of progress I suppose.

Little did I know that, thirty five years later, I’d be one of those on the hunt for that elusive last unicorn. Of the data science variety.

It seems like everyone is searching for their own kind of unicorn these days though. VCs want to grab the new Uber or AirBNB before the competition gets their paws on them. And in data science it’s become the go-to phrase to describe that mythical skill combination of all aspects of the dark arts wrapped up in a neat one-person meaty human package.

Data engineering. Data analysis. Software engineering. Data wrangling. Machine Learning. Artificial Intelligence. Data storytelling. Model building. Product management. People management. Soft skills. Hard skills. Tech skills.

SELECT * from above.

The Data Science Unicorn has to be proficient in all areas of data science to truly make the grade. (Or so a lot of job listings would have us believe.) Plus they have to hold at least a PhD. Possibly multiple Master’s Degrees as well. And have multiple side projects on their Github. And the right kind of internships and work experience on their resume before they even submit an application form.

And…and…and…and…

Is it any wonder so many young potential data scientists are getting turned off from the so-called “sexiest job of the 21st Century” before they’ve even left the safe bosom of their academic studies?

Where the rubber meets the road.

So how realistic is it to search for our own DS unicorns? And, more importantly, are they really what businesses need to deliver results from their data in the real world?

The modern data scientist has two options as their career progresses:

1. Specialist – go narrow and deep and really specialise on one particular area such as machine learning or data engineering,

or

2. Generalist – go broad and shallow and cover off more of the general bases.

The further you go in your career, the likelihood will be that you will eventually gravitate more towards option 1. There will be an inevitable attraction (and aptitude) towards one aspect of the profession over the others. This is perfectly normal and, due to the vastly different skill-sets needed for each different area, understandable.

The case for specialising – and where it makes sense.

Jeremie Harris of Sharpest Minds advises against going down the generalist route altogether. He has built up a strong working knowledge of what companies want through his work mentoring young data scientists and advises them to think first of the “kind” of data scientist they want to be.

I have a problem with this. At an early stage in their career they won’t have had enough exposure to the different areas of data science to really know which avenue they should concentrate on.

Does size matter?

Specialisation does, however, lend itself to working in very large corporate companies. They usually have the luxury of well resourced data science departments with very specific tasks for each team. If you have a data engineering team, an analyst team, a BI team and a modelling team, it makes more sense to look for new hires that fit into one of those specific pigeon holes.

They are also much more likely to be hiring you to look after a specific problem area they are facing so, of course, they’ll be more interested in a particular specialization.

That theory gets flipped on it’s head if you are running a smaller organisation, like a small company or early stage startup. With a small (or non-existent) data team, you’re much more likely to look for new hires with skill-sets that cover more of the bases at once. The luxury of hiring someone who “only” does machine learning engineering just isn’t available to you.

Full-stack. Fully loaded.

Which brings us back to option 2 – the Generalist.

My own experience of running an analytics team is very much on the small team with extremely wide remit side of things. I may be a tad biased on what I look for when recruiting because of this so best to make that clear up front. YMMV.

When you are running a team that covers business intelligence, ad-hoc reporting, analysis, regulatory reporting, model building and general data project troubleshooting and internal consulting – you can’t be picky about the salami slice of skills you might need from a team member.

On a day-to-day basis, you will need a small combination of THE LOT. Maybe not to a level that a skilled specialist who only concentrates on their own particular element offers but Just Enough Experience to Perform. That acronym of JEEP has to underlie everything you are hiring for when you never know what lies around the next corner.

Randy Au has a great quote in his article Succeeding as a data scientist in small companies/startups which sums up the “not quite knowing” what you’ll need the data team to do in your small company:

It’s usually quite likely that they don’t really have a full understanding of what they need. There’s just a generalized sense of “we have data, it seems useful, but we don’t have anyone who has the skills to make it useful.”

As Randy goes on to say, the main thing is being able to roll up your sleeves and make a difference on the ground TODAY. Then make sure that you’re building the right infrastructure to make full use of data/analytics/data science there TOMORROW. A machine learning engineer, regardless of their talent and smarts, just won’t be the right person for that particular job. You NEED a generalist.

When all is said and done.

When the shouting is all over though and the stack of CVs is sitting in front of me, I know exactly what I am looking for. Give me the analytics equivalent of the 1970s Dutch Total Footballer every time over the American Football special team kicker. Someone who can slot into any of these positions and not look too out of place. That suits my needs much more than a one trick pony, even if they are the best in the world at that one very special thing.

Then maybe, just maybe, over time and through experience, one of my Total Footballers can find they want to specialise and move on to a more suitable home for that stage of their career. Horses for courses. I’d be willing to wager that the other skills they’ll have picked up in the meantime will never be wasted even if they do go on to specialise and that, for me, makes it all the more reason to keep looking for my own unicorns wherever I might find them.