Don’t Waste Your Time On A Doomed Big Data Job

Ever wish you could get an easy answer to what is Big Data? Get the Big Data Checklist, totally free, along with weekly blog tips delivered directly to your inbox.

Time is the most precious thing we have. Spending time with the family, reading a book, playing video games, watching Netflix or whatever.

In my office there is my Les Paul desperately waiting for weeks now, to be picked up and played. It’s not that I don’t want to. Hell, I could go there, right now.

From experience I know that it would be a lot of fun. But I also know that in this moment, my time is better spent writing this post. So, as usual, I decide not to.

The time we spend on each activity is relatively short. That’s why these kind of decisions are easy.

To take a decision on a job however, that is on another level.

My motivation for this post is this: I want to make sure that you don’t waste your time on a job.

You spend a big chunk of your life at work. It would be a shame to waste it by joining the wrong project.

There are four key areas of any good big data project: Project management, data, people and budget.

You need to inspect these areas carefully before making a decision to join the project or the company.

Project Management

The project management of big data projects is very important.

In your interview, find out what the mission of the project is. The project should not be started like: “Big data is hip right now. So, lets do a big data project and see what we can do.”

Having no mission will lead just into chaos. The team will try to do do everything while accomplishing almost nothing.

“We choose to go to the Moon in this decade and do the other things, not because they are easy, but because they are hard”

John F. Kennedy, Sept. 1962

Now, that is what a mission looks like.

Go to the moon within the next eight years. The other things he mentions are build bigger rockets and do unmanned space exploration.

Going to the moon alone, a monumental task in the sixties.

When the mission is clear, find out the current timeline. The timeline tells you what needs to get done this year, to drive forward the mission.

The timeline should also include some milestones. Milestones let you plan your objectives that you want to achieve for the quarter.

Always remember define measurable objectives.

I know it sounds silly to mention that you need a mission, a timeline and milestones. Everybody does that. Right?

The truth is, that these basic things, mission, timeline, milestones, are often overlooked.

In September I attended the Big Data Minds conference in Berlin and spoke to many people.

Some of them had defined all the above and their business was thriving. Others were not given a concrete mission or a timeline.

Their life was not so good right now.

If you find out that there is no concrete mission to the project don’t even think about joining. Chances are high it will be a train wreck.


“Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it”

Dan Ariely

Dan Ariely sheds light on a major problem. Most companies have no freakin idea how to do a big data project.

But, you know what is worse than having no clue how to do a big data project?

A big data project that is not a big data project. People exaggerate how much data they actually have or will get.

I call these the “Not so big data projects”. There is a lot of data but you can process it with common tools.

You absolutely have to make sure you do not join such a project. Using big data technologies when you do not need them will almost certainly lead to failure.

You can ask these questions to make a prediction, if the project is a real big data project:

  • How fast is data coming in right now?
  • How much data do you have?
  • How much of the available data needs to be analyzed at once?
  • How fast do you think will the data grow in the future?

If it looks like big data, think about joining. If it’s not big data, walk away.


Try getting infos about the team. The best thing would be to meet them.

You need to find out if the right people are on board for doing big data. If there are colleagues who already have experience with big data and know what they are doing, that’s a good sign.

There should be at least a data scientist and a solution architect. The architect for building or choosing the platform. The data scientist for the analytics part.

By the way, there is the thinking that everything, from the platform to analytics solution, can be bought from a vendor. That does not work.

Only hiring external consultants to have success is futile.

Consultants can help a company tremendously to hit the ground running. By helping you to get the tools and processes for achieving your mission.

But what happens when they leave?

You need specialists who can carry on the work. Who can run and evolve the whole big data venture further.

Some great consultants won’t even show up if you do not have a capable big data team.

Neither should you.


The best case scenario is that the big data project has a realistic budget assigned. Not only for the current but for at least one or two years in advance.

In reality however, often times no concrete budget is assigned to the project. Or it is very low.

Why is having a proper budget so important?

Big data projects demand more resources than your average IT project. Because, you know, stuff is expensive and it also takes a lot of time in development.

Having a proper budget for one or two years in advance gives you the opportunity to plan ahead.

Imagine you have to fight for your budget every year! You have planned stuff, then a budget cut.

What a set back. Now you have to drop something.

It’s like that story from one of Michel Moore’s documentations. Some injured guy playing guitar has to chose which finger to keep. Because he can only afford to save one.

If you are right handed, better keep the one on the left hand and loose the one on the right. Either way, a fucked up decision to make.


There are a lot of other factors that influence your decision. Yes, money for instance, I know.

However, money isn’t everything. It’s about to find a job where you will spend your time meaningful.

It would make me very happy if, a few months from now, this post has helped you to make the right decision on a job.

I know one thing for sure: Tonight, I am going to play the hell out of this guitar!


Liked this post? Please share it with your peers! 🙂

Also, make sure to not to miss any new article by subscribing to the newsletter. This way I will be able to send you an email when I publish a new post



Comments 4

    1. Post
  1. Andrea’s you mentioned some of the questions to make a prediction if a project is a real big data project. May you provide some benchmark answers to gauge a big data project against?

Leave a Reply

Your email address will not be published. Required fields are marked *