All posts Coding Assessments

Writing Hidden Test Cases That Catch the Mistakes Students Actually Make

When a coding question is marked by its test cases, the test cases are the marking scheme. Four real student answers to one question, graded, show what each case is worth and how to write a set that catches the mistakes students actually make.

Book a free demo or try the 2-minute student test →

When a coding question is graded automatically, nobody reads the code to decide the mark. The test cases decide it. On Teemly, an answer that passes three of five cases earns three fifths of the question's marks, and every case counts for the same share.

So the test cases are the marking scheme. Writing them is not a technical chore to finish after the real question is written. It is the moment you decide what each mistake costs.

The clearest way to see that is to take one question, four answers students actually write, and run them.

One question, five test cases

Second largest number. Return the second largest distinct number in a list. Repeated values count once, so in [5, 5, 3] the second largest is 3. The list always has at least two distinct values.

The question has five test cases. Students can see the first three, inputs and answers, while they work. The last two are hidden: when a student runs their code, they are told only whether each hidden case passed or failed.

CaseInputExpectedSeen by student?
1[4, 1, 7, 3]4Yes
2[5, 5, 3]3Yes
3[10, 9]9Yes
4[-1, -5, -3]-3No
5[2, 8, 8, 6, 1]6No

Four answers, graded

These are four of the most common answers to this question. We ran each one through Teemly's grader, with the question worth 10 marks.

  • Answer A sorts the list and returns the second-last item.
  • Answer B walks through the list keeping a "largest" and a "second largest", both starting at zero.
  • Answer C removes the largest number once, then returns the largest of what is left.
  • Answer D removes the repeats first, then sorts and returns the second-last item.
AnswerVisible passedHidden passedMarks out of 10Where it failed
A: sort, take second-last2 of 31 of 26[5, 5, 3] and [2, 8, 8, 6, 1]: returned the repeated largest
B: start both at zero3 of 31 of 28[-1, -5, -3]: returned 0, a number not in the list
C: remove the largest once2 of 31 of 26The same two cases as A, for the same reason
D: remove repeats, then sort3 of 32 of 210-

Every answer run through the platform's own grader. Marks are the share of the five cases passed.

See this in your own lab. Teemly sets coding tests from your question bank, marks every program against test cases, and explains the results in plain English. We set it up free.

Book a free demo

What the table is telling you

Answer B is the reason hidden cases exist. It passes every case the student can see, so the student believes it is finished. Its mistake, starting from zero, only shows up when every number is negative, and none of the visible examples are. Without case 4, B scores full marks for code that returns a number that is not even in the list.

Answers A and C are caught by a visible case, and that is good. Case 2 shows the rule that repeats count once. A student who runs A sees case 2 fail, rereads the question and fixes it. A visible case that demonstrates a rule written in the question is teaching, not giving the answer away.

The marks are a decision you made by counting. A loses 4 marks because two cases test repeats. B loses 2 because one case tests negative numbers. If you think forgetting negative numbers deserves the same penalty as forgetting repeats, the fix is not to change anyone's mark. It is to add a second negative case.

The rule. Every case is worth the same share, so the number of cases testing a behaviour is that behaviour's weight. Decide what each mistake should cost, then write that many cases for it.

Start from the mistakes, not the inputs

The usual way to write test cases is to think of some inputs. The better way is to list the mistakes a student is likely to make, then write one case that each mistake would fail. Most of them come from a short list:

The mistakeThe case that catches it
Starting a running maximum or total at zeroEvery value negative
Forgetting that values can repeatThe largest or smallest value appearing twice
Off by one at the edge of a rangeThe smallest and largest values the question allows, such as counting primes up to exactly 2
Assuming there is always something in the listAn empty list, if the question allows one
Treating "Level" and "level" as differentMixed upper and lower case
Code that only handles the examplesAn input longer, or shaped differently, than any example
A correct but far too slow methodThe largest input the question allows, with a time limit set on the question

Not every question needs every row. A question about strings has no negative numbers. But going down this list before writing a single case is what turns a set of inputs into a marking scheme.

Four rules for a fair set

1. Stay inside your own question. This question promises at least two distinct values, so [5, 5] is not a fair case. A hidden case outside the stated limits does not catch a mistake. It punishes the student for believing what the question said.

2. Show the format, hide the idea. The visible cases should show one ordinary input and one that demonstrates any rule written in the question. The hidden cases test whether the student understood it.

3. Write expected answers as values. The grader compares values, not text. An answer of 32.0 matches an expected 32, a list written as [1, 2] matches [1,2], and spaces at either end are ignored. Words are compared exactly, though: "World hello" does not match "world hello".

4. Prove the set with a wrong answer. Solve the question correctly and check every case passes. Then write the mistake you most expect from a weak student, the way answer B starts at zero, and check that at least one case fails it. If your wrong answer passes everything, a case is missing. This is the same check as sitting the paper twice before a lab exam, and it takes ten minutes per question.

How many cases

For a lab question, three visible cases and three to five hidden ones is enough. Fewer than that and one case decides too much of the mark. Many more and the cases start testing the same mistake twice, which quietly doubles its weight.

Count the visible cases too. They are worth the same share as the hidden ones, so in our question the three visible cases are 6 of the 10 marks. A student who writes code that only works on the examples could keep most of the marks. If that is more than you intend, show fewer cases and hide more.

What the grader does, and what it does not

What it does:

  • Runs every case on every answer, identically, so a mark means the same in every section.
  • Splits the marks by the cases passed, so a nearly-right answer earns most of the question rather than zero.
  • Shows faculty which case each student failed. A class where half the students failed the negative-numbers case has one thing to revise, and it is not the whole topic.

What it does not:

  • Choose the cases. That judgement is yours, and it is most of the work.
  • Read the code. An answer that passes every case can still be badly written. That is what a viva, or a teacher's review, is for.
  • Weight one case above another. Weight comes from how many cases test a behaviour, which is why the counting above matters.

See it from the student's side

The sample test at teemly.in/try has two questions with hidden cases. Write a deliberately wrong answer and watch which case catches it; it is the fastest way to get a feel for what a good hidden case does.

For why marks that come from test cases tell a teacher more than a hiring platform's score does, see what a coding test built for hiring cannot tell a teacher. And when you are ready to write questions like this for your own department, that is what Teemly's coding assessments are built for.

Free PDF checklist

Not ready for a demo yet?

Get the Assessment Readiness Checklist: 42 questions to work through before you trust your next exam to a digital platform.

Get the free checklist
Free demo

Run your next coding lab on Teemly

We set it up for your department, free. Every program is marked against test cases, and Teemly AI explains the results in plain English.

Book a demo!