Unpredictable order of validation errors

Often, when creating web applications using Spring Boot, we need to ensure proper validation of incoming requests. This validation can check various criteria, such as the correctness of the data format, the presence of required fields, etc. However, in my Repository Finder project, I encountered an interesting problem related to validation and the order of errors. I will be happy to briefly describe it here.

To protect against invalid data in the Repository Finder, I implemented validation of incoming objects. I used the @NotEmpty and @NotNull annotations to check whether the user was sending a request with an empty username or completely empty JSON.

I used ControllerAdvice to handle validation errors in Spring Boot. I was creating a special class called ApiValidationErrorHandler that responded to MethodArgumentNotValidException exceptions and returned a response with validation errors.

@ControllerAdvice(assignableTypes = GithubController.class)
public class ApiValidationErrorHandler {

    @ExceptionHandler(MethodArgumentNotValidException.class)
    @ResponseBody
    @ResponseStatus(HttpStatus.BAD_REQUEST)
    public ApiValidationErrorResponseDto handleValidationException(MethodArgumentNotValidException e) {
        List<String> errorsFromException = getErrorsFromException(e);
        return new ApiValidationErrorResponseDto(errorsFromException, HttpStatus.BAD_REQUEST);
    }

    private List<String> getErrorsFromException(MethodArgumentNotValidException exception) {
        return exception.getBindingResult()
                .getAllErrors()
                .stream()
                .map(DefaultMessageSourceResolvable::getDefaultMessage)
                .collect(Collectors.toList());
    }
}

However, there was a problem in my tests. The tests assumed that errors would be returned in a specific order, which seemed logical. However, we found that the error order was unstable and changed depending on the test run. The probable cause was the use of Stream in the getErrorsFromException() method.

To solve this problem, I had to adapt my unit tests. Instead of relying on a specific order of errors, I started testing whether the error list contained the expected messages. In my test, it was enough to make a small change. It was necessary to replace:

.andExpect(jsonPath("$.errors[0]").value("Username can not be empty"))
.andExpect(jsonPath("$.errors[1]").value("Username can not be null"))

to:

.andExpect(jsonPath("$.errors", hasItems("Username can not be empty", "Username can not be null")))

Now the test does not depend on the specific order of errors, but checks whether the response contains all expected validation errors. This solution was more stable and allowed me to effectively test the validation of my requests.