Pilot projects: the missing step in your hiring process
A powerful, data-driven tool to help you make better hiring decisions and improve your candidate experience.
Eric Sanders
Eric Sanders
|
Product/Content Lead at Laskie
|
January 12, 2022

Let's think of the worst way to hire for a minute: Post a job description that's somehow both vague and overly specific on a website that charges you per click, receive hundreds of applications from people with experience totally unrelated to your job, manually pore though crazily-formatted resumes and generic cover letters ("Dear Sir and/or Madam") looking for a diamond in the rough who's at least worth interviewing, set up an interview and then both show up totally unprepared, attempt to ‘shoot the shit' and get a feel for them as a person without asking them any probing or insightful questions, use your intuition to decide that you like them and that they’re a good fit, pass them along to your colleague or manager for a second interview where they do the same thing, hire them, and pray for the best…

If you're laughing, it's because you've either done this or know people who still do. And it's not your fault—hiring is still stuck in an early-20th century paradigm of job posts, resumes, and interviews. The business model of the major job sites (caveat: I worked at Indeed for 3 years) is largely predicated on volume (clicks, job posts, messages, etc.), not quality—just like it was when employers placed an ad in the local newspaper in 1930. 

But today, with the ability to use smartly structured and personalized data on both sides of the matchmaking process, there's no need to think about hiring in the same way we used to. And this new, data-driven approach not only applies to finding candidates worth interviewing, but to vetting them as well.

Introducing pilot projects

Pilot projects (sometimes called “take-home projects”—my colleague Debbie coined the new name and we much prefer it here at Laskie) are simply brief, paid projects sent to finalists that mirror your actual job as accurately as possible. They’re the final step in the hiring process to help you differentiate top candidates from each other, as well as to give them a realistic preview of what the actual job entails so they can decide if they’re a good fit. In this way, they’re a win-win for employers and candidates. (Plus, of course, you always pay people fairly for these projects, so they’re usually appreciative even if you don’t hire them.)


What’s my obsession with pilot projects? In the past five years, I’ve created and administered hundreds of them for all kinds of roles—everything from auto-scored, 30-minute pilot projects for data entry candidates in the Philippines to 3-hour, manually-scored data analytics pilot projects for PhD I/O psychologists in Austin. 


But it’s also personal: I’ve been hired, twice—by the same founders, for two very different roles—by pilot projects. First, at Interviewed (YC S15), the assessments startup that was acquired by Indeed and became the largest job assessment platform in the world with 150MM+ users and 600,000 customers. And now, at Laskie, where I’m helping to change the job search paradigm from “generic resumes and job descriptions” to “personalized matchmaking”.


My career = thanks to pilot projects

In early 2017, between consulting gigs, I connected with Chris and Daniel (the co-founders of both Interviewed and Laskie) on a job board and told them: “I’m the perfect person for this job” (heading up the content team for Interviewed, the assessments startup). But my resume didn’t totally match my confidence, no matter how hard I tried to reword my work experience. So they said: “Sounds good—prove it.” They put me through the ringer with a pilot project to create an assessment for something I knew nothing about: financial planning. I barely took any breaks for an entire day, I was so engaged and excited.


Why was this such a meaningful experience for me? This project gave me a chance, for the first time, to show my future employer (not tell them) that I was the perfect person for the role. It also gave me a completely realistic preview of the actual job, which made me want it even more.


Beyond just being great for employers, well-designed pilot projects are a huge win for candidates, because they allow them to:

  • Showcase their skills in a job-specific setting
  • Get a realistic preview of the job, so they can decide if they really want it or not
  • Meet and briefly work with their future colleagues and managers
  • Be paid fairly for their work


Here’s what our CEO, Chris, has to say about them:

Structured interviews, work samples, and assessments are not enough

Structured interviews, where hiring managers ask all candidates the same set of questions—attempting to learn how a candidate really thinks, works, etc.—are a valuable early-stage tool to assess candidates in an ‘apples to apples' way. But since these questions are asked in the context of an often-awkward interview to candidates who may be nervous, their responses are more of a preliminary indicator of skill—not the best way to gauge a candidate's actual ability to do the job.

Beyond interviews, employers sometimes ask candidates to provide work samples—like a portfolio or other demonstrations of their past work. This makes sense for certain roles, like a designer, but falls short for most candidates, whose exact influence on any given project is hard to pinpoint. (Ask a marketing person to show you exactly what they did on their last company’s website, for example, and you’ll either get confusion or exaggeration.) Plus, even if a candidate can provide work samples that are entirely their own actual work, those samples are unlikely to be particularly relevant to your job, and you’ll have to do a lot of inferring to decide if those skills are transferable to your specific needs.

Lastly, there are job assessments. These can range from psychological tests that attempt to measure alignment between a candidate and a particular type of job to cognitive or hard skill tests that purport to measure more-objective constructs like “Logical Reasoning” or even “Microsoft Excel”. Assessments can be a valuable early-stage tool for certain roles to weed out totally unqualified candidates, but they are often overused (too many tests, too long, redundant questions, etc.) and can be borderline offensive to more qualified or senior applicants, who may drop out of the hiring process rather than take another irrelevant and unpaid assessment. Even at their best, assessments are generic and do not give candidates the opportunity to showcase how their skills are relevant to your particular job.

Pilot projects to the rescue

So what’s the best way to tell if someone can do the actual job? How about… having them do the actual job? It almost sounds too obvious to be true—but as soon as you start administering pilot projects for all your roles, you’ll kick yourself and ask, “How was I not doing this all along?”

Ready to get started? Here are some key principles you can use to start brainstorming and designing your own pilot projects:

Send them to finalists only. Only send your pilot project to actual finalists, since these projects require a time and energy investment on both sides. Sending them to all candidates will only serve to ruin your reputation and make you a company no one wants to interview with. Use the pilot project as a differentiator only for people you are seriously considering hiring, not as a preliminary screening tool.

Use them for every job. Pilot projects aren’t just for software engineers. Whether you’re hiring in engineering, product, design, marketing, sales, or beyond, there’s no job that won’t be well-served by a pilot project. Again, I’ve used them extensively to vet even part-time data entry candidates, and the results have been consistently astonishing—saving time, money, and the headaches of making a bad hire.

Value candidates’ time. A typical pilot project should take candidates 1-3 hours (ideally uninterrupted) in a 24-hour time period. More time might be okay for more-complex roles, but be prepared to pay accordingly, and don’t be surprised if some candidates drop out.

Pay people fairly. It's essential to pay people fairly for pilot projects—it makes your company look good (aka professional) and also helps the person take off from work, prioritize your project, etc.

Make it realistic. You want your pilot project to be as close to a microcosm of the actual job as possible—so use your real business data, use cases, and examples for inspiration. Making the project ultra-realistic helps you assess candidates more accurately, and also lets them understand exactly what the role entails (as well as who they’ll be working for/with), which helps them self-select if they're a good fit or not.

Focus on key job tasks. I’ll discuss this in more depth in the next post, but, to get started, try to break your job down into 35 key “job tasks” (fewer is better). These will provide the foundation for what you’ll have candidates do in the pilot project. For example, when I was applying to Interviewed, the actual key job task (what I’d be doing in the job) was to “create assessments”—so the majority of my pilot project (and final score) centered around actually creating an assessment. It really is that simple.


Have a clear plan for scoring. Decide in advance who on your team will be grading the pilot projects and, at a high level, how their scores will be weighted (e.g. should the manager's opinion have the same weight as the colleagues?). You’ll need to use the same grading approach for each candidate to make sure you’re comparing ‘apples to apples’. (No changing the rules just because you like a candidate.)

There’s a lot more to say on this topic—in part two, I’ll dive much deeper into how to create a good pilot project, with specific examples of how we design and use them in our own hiring processes here at Laskie.

In the meantime, I can promise you this: the best way to hire people who will a) do the job well, b) like the job, and c) stick around is to have them do a pilot project first.

Looking for work?

Visit Laskie and complete our short application process to get matched (for free) with full-time jobs, freelance work, and moonlight gigs that meet your requirements.


Hiring?

Get started on Laskie to get matched with qualified, pre-vetted candidates who are actively looking for work or to use our elastic talent acquisition services.

About Author
Eric Sanders
Eric Sanders
Product/Content Lead at Laskie