When a coding question is graded automatically, nobody reads the code to decide the mark. The test cases decide it. On Teemly, an answer that passes three of five cases earns three fifths of the question's marks, and every case counts for the same share.
So the test cases are the marking scheme. Writing them is not a technical chore to finish after the real question is written. It is the moment you decide what each mistake costs.
The clearest way to see that is to take one question, four answers students actually write, and run them.
One question, five test cases
The question has five test cases. Students can see the first three, inputs and answers, while they work. The last two are hidden: when a student runs their code, they are told only whether each hidden case passed or failed.
| Case | Input | Expected | Seen by student? |
|---|---|---|---|
| 1 | [4, 1, 7, 3] | 4 | Yes |
| 2 | [5, 5, 3] | 3 | Yes |
| 3 | [10, 9] | 9 | Yes |
| 4 | [-1, -5, -3] | -3 | No |
| 5 | [2, 8, 8, 6, 1] | 6 | No |
Four answers, graded
These are four of the most common answers to this question. We ran each one through Teemly's grader, with the question worth 10 marks.
- Answer A sorts the list and returns the second-last item.
- Answer B walks through the list keeping a "largest" and a "second largest", both starting at zero.
- Answer C removes the largest number once, then returns the largest of what is left.
- Answer D removes the repeats first, then sorts and returns the second-last item.
| Answer | Visible passed | Hidden passed | Marks out of 10 | Where it failed |
|---|---|---|---|---|
| A: sort, take second-last | 2 of 3 | 1 of 2 | 6 | [5, 5, 3] and [2, 8, 8, 6, 1]: returned the repeated largest |
| B: start both at zero | 3 of 3 | 1 of 2 | 8 | [-1, -5, -3]: returned 0, a number not in the list |
| C: remove the largest once | 2 of 3 | 1 of 2 | 6 | The same two cases as A, for the same reason |
| D: remove repeats, then sort | 3 of 3 | 2 of 2 | 10 | - |
Every answer run through the platform's own grader. Marks are the share of the five cases passed.
See this in your own lab. Teemly sets coding tests from your question bank, marks every program against test cases, and explains the results in plain English. We set it up free.
Book a free demoWhat the table is telling you
Answer B is the reason hidden cases exist. It passes every case the student can see, so the student believes it is finished. Its mistake, starting from zero, only shows up when every number is negative, and none of the visible examples are. Without case 4, B scores full marks for code that returns a number that is not even in the list.
Answers A and C are caught by a visible case, and that is good. Case 2 shows the rule that repeats count once. A student who runs A sees case 2 fail, rereads the question and fixes it. A visible case that demonstrates a rule written in the question is teaching, not giving the answer away.
The marks are a decision you made by counting. A loses 4 marks because two cases test repeats. B loses 2 because one case tests negative numbers. If you think forgetting negative numbers deserves the same penalty as forgetting repeats, the fix is not to change anyone's mark. It is to add a second negative case.
Start from the mistakes, not the inputs
The usual way to write test cases is to think of some inputs. The better way is to list the mistakes a student is likely to make, then write one case that each mistake would fail. Most of them come from a short list:
| The mistake | The case that catches it |
|---|---|
| Starting a running maximum or total at zero | Every value negative |
| Forgetting that values can repeat | The largest or smallest value appearing twice |
| Off by one at the edge of a range | The smallest and largest values the question allows, such as counting primes up to exactly 2 |
| Assuming there is always something in the list | An empty list, if the question allows one |
| Treating "Level" and "level" as different | Mixed upper and lower case |
| Code that only handles the examples | An input longer, or shaped differently, than any example |
| A correct but far too slow method | The largest input the question allows, with a time limit set on the question |
Not every question needs every row. A question about strings has no negative numbers. But going down this list before writing a single case is what turns a set of inputs into a marking scheme.
Four rules for a fair set
1. Stay inside your own question. This question promises at least two distinct values, so [5, 5] is not a fair case. A hidden case outside the stated limits does not catch a mistake. It punishes the student for believing what the question said.
2. Show the format, hide the idea. The visible cases should show one ordinary input and one that demonstrates any rule written in the question. The hidden cases test whether the student understood it.
3. Write expected answers as values. The grader compares values, not text. An answer of 32.0 matches an expected 32, a list written as [1, 2] matches [1,2], and spaces at either end are ignored. Words are compared exactly, though: "World hello" does not match "world hello".
4. Prove the set with a wrong answer. Solve the question correctly and check every case passes. Then write the mistake you most expect from a weak student, the way answer B starts at zero, and check that at least one case fails it. If your wrong answer passes everything, a case is missing. This is the same check as sitting the paper twice before a lab exam, and it takes ten minutes per question.
How many cases
For a lab question, three visible cases and three to five hidden ones is enough. Fewer than that and one case decides too much of the mark. Many more and the cases start testing the same mistake twice, which quietly doubles its weight.
Count the visible cases too. They are worth the same share as the hidden ones, so in our question the three visible cases are 6 of the 10 marks. A student who writes code that only works on the examples could keep most of the marks. If that is more than you intend, show fewer cases and hide more.
What the grader does, and what it does not
What it does:
- Runs every case on every answer, identically, so a mark means the same in every section.
- Splits the marks by the cases passed, so a nearly-right answer earns most of the question rather than zero.
- Shows faculty which case each student failed. A class where half the students failed the negative-numbers case has one thing to revise, and it is not the whole topic.
What it does not:
- Choose the cases. That judgement is yours, and it is most of the work.
- Read the code. An answer that passes every case can still be badly written. That is what a viva, or a teacher's review, is for.
- Weight one case above another. Weight comes from how many cases test a behaviour, which is why the counting above matters.
See it from the student's side
The sample test at teemly.in/try has two questions with hidden cases. Write a deliberately wrong answer and watch which case catches it; it is the fastest way to get a feel for what a good hidden case does.
For why marks that come from test cases tell a teacher more than a hiring platform's score does, see what a coding test built for hiring cannot tell a teacher. And when you are ready to write questions like this for your own department, that is what Teemly's coding assessments are built for.