An evaluation set is a collection of questions together with the expected passages or answers, established in advance. It lets you objectively measure the quality of search and compare settings, instead of judging from a few random tries.
Example
{"question":"Cooling-off period?","expected_passage":"cgv#art7"}