The Evil of Technical Interviews
I've been on both sides of technical interviews in the past, and whichever side I am on, I have realized that they are about worthless. Worth than worthless, actually, because they can lead to one rejecting string candidates, and accepting weak ones.
First, let's look at the alternatives. The best way to judge someone's technical ability is to work with them. This isn't usually possible before you hire them, and even if it is, it takes a lot of time. You can hold non-technical interviews, and simply ask about someone's experience, etc. The advantage here is that even a non-technical person can do a non-technical interview. The disadvantage is that some people fake it, and put lots of buzz words and trumped up experience on their resumes. We like to believe that if someone's been working at a reputable company in a certain role for years, that they're good at it, but that sadly isn't true sometimes. What's more, their bosses might give them a good reference just to get rid of them.
The technical interview is seen as the solution to this problem. You ask a bunch of "real" technical questions that the candidate can't possibly fake. They have to know it, or not. It seems simple: If they know most of the answers, they are judged to be a strong candidate. If not, they aren't. Unfortunately, it doesn't work that way.
The first problem that technical interviews have is that they are essentially standardized tests. Standardized tests serve a purpose, and they are better than nothing, but they are far, far from perfect. I know people who can speak perfectly fine conversational English who have "failed" the TOEFL test. At the same time, many people who are good at taking tests can get high scores on the TOEFL test, but have little practical command of English. It would be far better to sit students down and talk to them for two hours in English to actually judge their ability, than to accept their TOEFL scores on faith. At least the questions on TOEFL are standardized though. (That's part of why the tests can be gamed). Technical interviews aren't very standardized, and the people giving them may or may not actually be experts.
Back to the standardized tests, when I was in high-school I participated in a nation-wide computer contest, twice. Both times I was frustrated because the questions were obviously made up by someone who knew nothing about computers, but had checked a bunch of computer books out of the library to make up these questions. Few of them were technically *wrong*, but many were ambiguous, and many of them weren't relevant to ... anything. I remember one question was:
Q. What does POST stand for?
A. Power On Self Test
Seriously? I just happened to remember that from a text-book at school. I've never used the term. I've never seen it used anywhere else. Not during 4 years of studying computer science. Not working as a "Senior Programmer Analyst" at a large company. Not working as a computer consultant. But since someone decided to put that on the test, I could have lost a point. (I actually knew that one, but that doesn't affect my point here). Even on all the movies with totally random BS computer stuff, I haven't heard that term mentioned. There were lots of stupid questions like that, while they left stuff that actually mattered off. Any time I complained about it, people just said "Well you just don't know the answers, what's all... don't be a sore loser." Riiight. Well I placed at the local and national level, so nobody can say that anymore.
But that was just a silly contest, it didn't really matter that much who won or lost. Something that affects which employee you will hire (or which job you can get) is a little more important than a high-school contest.
One thing I learned the hard way early-on doing interviews is that interviewees often have cheat-sheets. Since often the similar questions are asked, interviewees often have cheat sheets. For SAP jobs, you can just go look on SDN and type "Interview questions". I am sure they are out there for other types of cheat sheets as well.
More than that, some people tend to "learn" by memorizing things rather than actually understanding them. These are the last kind of person you want working on your job, but they are also the type most likely to pass a technical interview. Chinese and Indians are (in general, of course) especially and used to studying this way, and thus good at passing tests, and by extension technical interviews.
Concrete Examples:
False Negatives
The problem is, that the questions tend to favor a memorization approach, while the actual work doesn't, at all. For example, if I ask someone an SAP question like "What is the transaction code for the ABAP Editor", the answer should be "Who cares?" Yes, it's SE38 (or se80 or a host of other similar transactions), but the point is that even if you forgot, you could look through the menus and find it. You could also look it up on the internet in 10 seconds. Yet I've had interviews where people asked me things like "What is the transaction code for..." and asked about some obscure function I haven't used in 4 years.
Of course it's not just SAP, for those who are into Java or C, the same thing holds. If I asked "What parameters does the foobar() function take?", well it doesn't matter. Many programmers are used to using the CodeSense style popups that appear in the IDE and help you out there, so there's no point in memorizing the parameters. I programed in Delphi for years, but I couldn't tell you the parameter lists for many functions - but it doesn't mean I couldn't still code in Delphi. You can have a lot of smart and intelligent people who understand computers inside and out, but who don't memorize trivial things. Anyone who's graduated from a decent college in Computer Science probably knows what they are doing, but may easily flunk these kinds of questions - even in something they have years of experience in. What's more, learning a new computer language is not too difficult, but you will start not knowing the keywords and library methods. That's not where people's skill lies.
False Positives:
Perhaps the bigger problem is false positives. If you yourself miss out on a job opportunity, that's sad, but you will have another chance. If you hire someone who doesn't know what they're doing, you may be stuck working with them for a long time.
When I first started conducting technical interviews, I was relatively easy on people. I mainly looked at their experience and decided they should be ok, and then asked a few simple questions I figured they would get right if they weren't retarded. They almost always did. In fact, I found I would ask lots and lots of memorization type questions, and they would usually get them right with no problems. I didn't realize it was because of the "memorize trivial facts" style of learning, in combination with the above mentioned cheat-sheets.
After approving people and seeing the paper thin understanding they actually had of what they were doing, I realized how bad this interview approach was. I've had people tell me they "fixed the bug", when all they did was comment out the lines causing the problem. Yeah, they fixed the bug... but that's like curing a cold with a bullet to the brain. I realized people may memorize function and event names, transaction codes, and even parameter signatures, but not have the slightest idea of how to thing logically or analyze code. One day there was a problem that a team of three people I had hired couldn't solve. I looked at it, and it wasn't that difficult if you understood what it was trying to do, so... I fixed it. But now, I like to send this (and similar samples) to people I interview and ask them to tell me what it's doing, what the bug is, and how to fix it. (It was a string processing routine about 10 lines long).
Conclusion
At any rate, the best method is to have someone come in as a co-op or intern, then you can work with them and decide whether to hire them or not. If they are an experienced hire, you could have them come in for a project or two on a contract basis. If they are good, you can offer them a job. This is also good for employees, because you might receive a chance you otherwise wouldn't have.
The method of testing with code samples I mentioned above is also good, but it does have some weaknesses. Someone may be good in general, but not able to figure your particular sample out. Worse yet, if you use the same sample a lot, it will probably show up on cheat sheets somewhere, and people will memorize that too. The main point here is that there is no real correlation between memorizing things, and being able to think with logical clarity about algorithms and how to break down complex tasks into simple ones. The latter is very important, the former isn't so much.
Unfortunately, for those taking interviews, there's not much to be done except play along. For those of us giving interviews, however, we have a chance to tilt the playing field in the right direction at least a little bit.
First, let's look at the alternatives. The best way to judge someone's technical ability is to work with them. This isn't usually possible before you hire them, and even if it is, it takes a lot of time. You can hold non-technical interviews, and simply ask about someone's experience, etc. The advantage here is that even a non-technical person can do a non-technical interview. The disadvantage is that some people fake it, and put lots of buzz words and trumped up experience on their resumes. We like to believe that if someone's been working at a reputable company in a certain role for years, that they're good at it, but that sadly isn't true sometimes. What's more, their bosses might give them a good reference just to get rid of them.
The technical interview is seen as the solution to this problem. You ask a bunch of "real" technical questions that the candidate can't possibly fake. They have to know it, or not. It seems simple: If they know most of the answers, they are judged to be a strong candidate. If not, they aren't. Unfortunately, it doesn't work that way.
The first problem that technical interviews have is that they are essentially standardized tests. Standardized tests serve a purpose, and they are better than nothing, but they are far, far from perfect. I know people who can speak perfectly fine conversational English who have "failed" the TOEFL test. At the same time, many people who are good at taking tests can get high scores on the TOEFL test, but have little practical command of English. It would be far better to sit students down and talk to them for two hours in English to actually judge their ability, than to accept their TOEFL scores on faith. At least the questions on TOEFL are standardized though. (That's part of why the tests can be gamed). Technical interviews aren't very standardized, and the people giving them may or may not actually be experts.
Back to the standardized tests, when I was in high-school I participated in a nation-wide computer contest, twice. Both times I was frustrated because the questions were obviously made up by someone who knew nothing about computers, but had checked a bunch of computer books out of the library to make up these questions. Few of them were technically *wrong*, but many were ambiguous, and many of them weren't relevant to ... anything. I remember one question was:
Q. What does POST stand for?
A. Power On Self Test
Seriously? I just happened to remember that from a text-book at school. I've never used the term. I've never seen it used anywhere else. Not during 4 years of studying computer science. Not working as a "Senior Programmer Analyst" at a large company. Not working as a computer consultant. But since someone decided to put that on the test, I could have lost a point. (I actually knew that one, but that doesn't affect my point here). Even on all the movies with totally random BS computer stuff, I haven't heard that term mentioned. There were lots of stupid questions like that, while they left stuff that actually mattered off. Any time I complained about it, people just said "Well you just don't know the answers, what's all... don't be a sore loser." Riiight. Well I placed at the local and national level, so nobody can say that anymore.
But that was just a silly contest, it didn't really matter that much who won or lost. Something that affects which employee you will hire (or which job you can get) is a little more important than a high-school contest.
One thing I learned the hard way early-on doing interviews is that interviewees often have cheat-sheets. Since often the similar questions are asked, interviewees often have cheat sheets. For SAP jobs, you can just go look on SDN and type "Interview questions". I am sure they are out there for other types of cheat sheets as well.
More than that, some people tend to "learn" by memorizing things rather than actually understanding them. These are the last kind of person you want working on your job, but they are also the type most likely to pass a technical interview. Chinese and Indians are (in general, of course) especially and used to studying this way, and thus good at passing tests, and by extension technical interviews.
Concrete Examples:
False Negatives
The problem is, that the questions tend to favor a memorization approach, while the actual work doesn't, at all. For example, if I ask someone an SAP question like "What is the transaction code for the ABAP Editor", the answer should be "Who cares?" Yes, it's SE38 (or se80 or a host of other similar transactions), but the point is that even if you forgot, you could look through the menus and find it. You could also look it up on the internet in 10 seconds. Yet I've had interviews where people asked me things like "What is the transaction code for..." and asked about some obscure function I haven't used in 4 years.
Of course it's not just SAP, for those who are into Java or C, the same thing holds. If I asked "What parameters does the foobar() function take?", well it doesn't matter. Many programmers are used to using the CodeSense style popups that appear in the IDE and help you out there, so there's no point in memorizing the parameters. I programed in Delphi for years, but I couldn't tell you the parameter lists for many functions - but it doesn't mean I couldn't still code in Delphi. You can have a lot of smart and intelligent people who understand computers inside and out, but who don't memorize trivial things. Anyone who's graduated from a decent college in Computer Science probably knows what they are doing, but may easily flunk these kinds of questions - even in something they have years of experience in. What's more, learning a new computer language is not too difficult, but you will start not knowing the keywords and library methods. That's not where people's skill lies.
False Positives:
Perhaps the bigger problem is false positives. If you yourself miss out on a job opportunity, that's sad, but you will have another chance. If you hire someone who doesn't know what they're doing, you may be stuck working with them for a long time.
When I first started conducting technical interviews, I was relatively easy on people. I mainly looked at their experience and decided they should be ok, and then asked a few simple questions I figured they would get right if they weren't retarded. They almost always did. In fact, I found I would ask lots and lots of memorization type questions, and they would usually get them right with no problems. I didn't realize it was because of the "memorize trivial facts" style of learning, in combination with the above mentioned cheat-sheets.
After approving people and seeing the paper thin understanding they actually had of what they were doing, I realized how bad this interview approach was. I've had people tell me they "fixed the bug", when all they did was comment out the lines causing the problem. Yeah, they fixed the bug... but that's like curing a cold with a bullet to the brain. I realized people may memorize function and event names, transaction codes, and even parameter signatures, but not have the slightest idea of how to thing logically or analyze code. One day there was a problem that a team of three people I had hired couldn't solve. I looked at it, and it wasn't that difficult if you understood what it was trying to do, so... I fixed it. But now, I like to send this (and similar samples) to people I interview and ask them to tell me what it's doing, what the bug is, and how to fix it. (It was a string processing routine about 10 lines long).
Conclusion
At any rate, the best method is to have someone come in as a co-op or intern, then you can work with them and decide whether to hire them or not. If they are an experienced hire, you could have them come in for a project or two on a contract basis. If they are good, you can offer them a job. This is also good for employees, because you might receive a chance you otherwise wouldn't have.
The method of testing with code samples I mentioned above is also good, but it does have some weaknesses. Someone may be good in general, but not able to figure your particular sample out. Worse yet, if you use the same sample a lot, it will probably show up on cheat sheets somewhere, and people will memorize that too. The main point here is that there is no real correlation between memorizing things, and being able to think with logical clarity about algorithms and how to break down complex tasks into simple ones. The latter is very important, the former isn't so much.
Unfortunately, for those taking interviews, there's not much to be done except play along. For those of us giving interviews, however, we have a chance to tilt the playing field in the right direction at least a little bit.
Comments
Post a Comment