Online Coding Tests - The Holy Grail of Software Recruitment ?

Mary, an experienced civil engineer applies for a structural engineer’s job at  Well Reputed Corporation (WRC). Before considering her for a first round interview,  WRC wants to gives a test to assess her skills in structural design for high rise buildings.

Under 3 hours, the test requires her to design and build a small structure to house a hamster. The house has to be one story, with several rooms and two entry doors. With the basic equipment given to her (a manual saw & screw driver), Mary works hard to get the task done. While a hamster house does not need her specialized structural design skills she does takes some shortcuts, which she knows are wrong, in order to complete the task in time. The hiring team at WRC puts a hamster inside the house and measures for how long the hamster stays inside. Unfortunately for Mary, the hamster exits the house in 23 seconds, while the minimum criterion to go to the next round of interview was 60 seconds.

Fortunately for civil engineers, such tests in the hiring process are quite rare. Unfortunately for software developers, the equivalent of this test in programming are increasingly becoming the norm in hiring.

The advent of web sites like codility.com, hackerrank.com, testdome.com etc. has made is easy for recruiters to evaluate software development candidates using a web based coding test. The recruiter sends a link to the candidate who spends some hours to give the test, results are automatically evaluated online through pre-developed unit tests and given to the recruiter in form of a numeric score. Based on some threshold, the numeric score makes it trivial to decide the future of the candidate’s resume (trash or prospective hire). Humans who write computer programs in their daily lives are themselves evaluated by a program. Is this a win-win for everyone?

Beyond doubt, for recruiters these web sites are a savior. Without a good technical background they can determine the technical proficiency through an actual test, rather than what the candidate wrote in her resume. For candidates applying to a programming job, it brings some transparency to the interview process while evaluating the core skill they are being hired for. However the question arises if this recruitment method is fool proof and does it actually evaluate the skills a candidate needs to be successful at her job?

Can good software developers be filtered in by having them demo their skills in a pressure environment under conditions which they will rarely face in any professional software company? On top of that, they are given basic tools like a barely functioning Integrated Development Environment (IDE) with limited or no IntelliSense and auto-correction capabilities. While finding the highest number in a matrix or finding if a number is power of 4 are genuine algorithms problems, from an interviewing standpoint they are nothing more than interesting puzzles. These puzzles are a small fraction of the problems a professional developer faces in her career.

Modern software systems are composed of multiple layers with visualization, business logic and data forming the high-level trilogy. The puzzles that web based coding tests ask are a small component forming these layers and they generally reside inside the business logic. Building the layers, allowing easy scalability in each, while allowing for easy integration between them and fixing bugs are some of the more serious challenges a developer faces. And so far there has been no demonstrable evidence or an empirical scientific study that shows a co-relation between algorithm puzzle solving and designing scalable and modular systems.

One a side note, the name and advertising of these programming sites also leave much to be desired. For example, the company codefights.com advertises itself as a platform to showcase one’s programming skills, based on which one gets selected by an employer. Are software developer’s gladiators demonstrating their skills in an online Roman Colosseum, to get the attention of their masters (hiring companies)? These names will be considered borderline unprofessional in other fields (can you imagine a recruiting site www.civilengineerfights.com ?). Maybe the next step is an online skills demonstration platform called http://www.codersgonewild.com (Wait! this domain is already taken, though programmersgonewild.com is still available for any takers). Just like with few other things in the profession of software development some degree of professional immaturity is tolerated and in fact glorified in certain organizations. This is all done in the garb of being cool, or changing the world (because any other objective is too low when creating software!).

In the absence of evidence showing a co-relation between algorithm-puzzle (or puzzle?) solving and designing scalable and modular systems, why do hiring companies insist on a web-based automated programming test? Apart from the fact that an automated system results in savings of effort and time, these tests are simple to conduct and verify. From a mathematical standpoint, the algorithms given in the puzzles are closed form (like an equation) and give deterministic outputs whose accuracy can be easily verified in polynomial time. For example, to verify an algorithm that predicts whether a given number is power of 4 or not, you just pass it some numbers which are powers of 4 and some that aren’t, and see if the algorithm gives the correct yes/ no answer. After the tests are run, the recruiter just sees the final result which is a number whose value is based on the number of verification tests the algorithm passed.

The verification tests are also varied in nature, checking for true/false positives & negatives, with some also checking the run time of the algorithm. However, verification of run-time is not accurate and is quite arbitrary since the accuracy depends on the background threads and exact number of context switches they cause. This requires sophisticated isolation of processes which most web based coding are unable to do due to technical reasons. Hence timing tests are as arbitrary as a hamster deciding how long to stay in the house.

Another significant problem with these coding tests is the possibility of mastering the puzzles with practice. Books like the widely popular ‘Cracking the Coding Interview: 150 Programming Questions and Solutions’ by Gayle Laakmann McDowell aim to ‘crack the code’ and help developers to become experts at these puzzles. Hence, there is a good chance that companies using these puzzles to hire developers may not necessarily be hiring the best brains as they intended to, but those who have spent hours practicing online. Rather than rewarding productive activities like contributing to open source or learning new technologies, companies are encouraging career growth by practice in puzzle solving.

So even though algorithm puzzles are easy to evaluate, the downside is their degree of arbitrariness as well as their mastery after only a few months of practice. On the other side, no self-help books can teach design and debugging skills as these cannot be mastered in a few months. Unless she is a prodigy, every developer has to pay the dues to master these skills by countless hours of actual programming and deep understanding of concepts. So why don’t recruiters use online tests that evaluate software design/debugging skills? Unfortunately, the applications of these skills is dependent on the problem context and other factors, making it open ended in nature. Open ended problems are difficult for a computer program to analyze, and no trustworthy automated test exists to evaluate them. Maybe machine learning will make it possible in the future, but till then evaluating the quality of a design will require human intervention, and an expert one at that. Arranging the human intervention aspect takes effort and requires more organization resources. Therefore they are not very appealing to recruiters especially when an automated approach that looks good on paper, is available.

So what can be done to remedy the situation? A solution is to conduct the coding test, but allow human intervention for evaluating the results of the tests and not to go by a numeric score alone. If the solution is incomplete allow for the possibility that the developer was focusing on good design or doing a test first development, and did not get sufficient time to complete the working solution. Perhaps the candidate was not familiar with the development environment (remember, it is always good to give candidates all the tools they are used to) or maybe the candidate buckled under the unrealistic time pressure and made errors.

Realistic time pressures, which exist in all software groups are measured in days or weeks. Unless you are tasked with coding and uploading a virus into an alien ship to save the world, in majority of jobs there are no catastrophic consequences if your code is not production ready in 3 hours. Hence, it is very much possible to find a gem in the author of an incomplete coding test.

A similar approach can be used to evaluate debugging skills. A buggy code can be given to the developer and her approach to debug the issue carefully observed. Even though a developer may not be able to identify and solve all the issues in code, the approach taken by the developer can give a very good idea of her debugging skills.

While candidates should definitely be given coding exams, the code should be evaluated in a more holistic way by looking at the design and the thought process of the candidate. By taking the harder approach of evaluating the design and debugging skills of a candidate, recruiters are guaranteed to get a well-rounded candidate compared to using an automated coding test, which in the best case will filter in an algorithm expert and in the worst case a candidate whose only expertise is in cracking coding tests.

Love or hate this article ? Have other ideas ? Please leave a comment below !