I agree with you that the author had some really crazy preferences in his example. Penalizing a front end webdev who aren't on LinkedIn as harshly as you penalize those who don't know HTML is absolutely bonkers!
In his defense, I think it was just meant to be an arbitrary example. You could avoid this problem by dramatically lowering the point values assigned to college degrees and LinkedIn profiles and upping the credit assigned to having Javascript skills.
The real problem is that there's no distinction made between someone such as myself who knows a bit of Javascript and someone who is an expert in it. Some criteria are binary, others are not.
It is binary, mainly because it has to be. You couldn't reliably job out a screening like this if it required the person to understand the nuances of JS, or any other skill requiring domain expertise.
But as long as you weight it properly, any candidate that makes the short list has javascript as competency. Their level of expertise can be teased out as you go. This is meant to be a first step to go from 300 down to 20, not to take you all the way there.
There are other ways to make it more nuanced - identifying keywords that would make it more likely to identify a JS expert in a resume or cover letter (not a replacement for looking at someone's code, but again something a non programmer could do). Or looking at someone's github account and seeing if they have JS repos (maybe factoring in the # of people following those repos). The formula would need some tweaking, but those are objective measurements that could get you closer.
The point of the system is to enable someone who doesn't have your knowledge to do the initial screening with a high degree of accuracy. The weighting of the criteria is usually what prevents someone from getting tossed for the wrong reasons.
- The college degree example is solved by weighting, as I mentioned. I agree with you - we've hired plenty of people in the past who didn't have a degree. That's why it would be considered secondary criteria - if you do it right they won't get heavily penalized for not having a degree if that's not important to you.
- LinkedIn, portfolios, etc. - the point is again what's important to you. For example, not giving me a portfolio is a deal breaker. I have hundreds of resumes to look through, and if I can't easily see your work I don't have time to hunt for it. It says at a minimum you didn't take the time follow instructions in the posting.
- The third example is your strongest argument. It can also be solved by weighting though, by giving more difficult technical skills more weighting. I know plenty of people who look for folks that know Java because it's relatively easy for those folks to pick up Ruby.
You're right - a few diamonds might slip through the cracks. But in most cases I think you'd miss them anyway.
This site may already exist. Identified (www.identified.com) assigns a score on each candidate's profile based on their network, previous experience, and education. Recruiters search for a set of criteria and are presented with list of relevant candidates sorted by score.
This is really no different from the HR filters many large companies have that result in people asking questions like "I see you have 6 years of Java experience. Do you have any J2SE experience?". The only possible difference is weighting of criteria and I'm not convinced HR departments don't do that already.
What a screening system like this does in practice is save you from having to look at hundreds of resumes from people who aren't even remotely qualified - the folks who carpet bomb their resume out to anyone.
You're the one creating the criteria and determining the weighting, so if the system excludes people because they don't have J2SE experience it's only because you put it on the list and made it extremely important.
What it does in practice is encourage keyword-stuffing.
I've been going through dozens of iOS and webdev resumes a week and about two in three have the right keywords. Their skills don't correlate strongly with those keywords. What has gotten dramatically better results for me has been this: put programmers through some automated programming tests (kind of like an in house version of interviewstreet.com), and filter any graphic artists / designer entirely based off of their portfolios.
That has quickly weeded out those who can't code (or design).
Interview street looks fantastic. You probably don't need a system like this for dev hires with a tool like that.
And you're right about designers - I look at portfolios more than anything else. This system does save me having to look at resumes to figure out whether they have a portfolio or not. It at least weeds out that third :)
Fail. You'll miss someone who lacks some minor criteria but is stellar in other areas.
"No college degree." Toss. You'll miss a bunch of the really good people I know who didn't finish.
"No LinkedIn profile." Toss. You'll miss good people who honestly don't care about LI.
"Expert in JS, not so much in CSS." Toss. You just lost a JS guru who's smart enough to pick up the other things you need, as you need them.
Unless you filter carefully, this step will filter out the quirks, probably the creative, driven people you really want.