data provided by an user is somehow incorrect and could cause my domain object to get into invalid state. So a subclass of DomainException extends RuntimeException is thrown.
Perfect.
Now: how should I handle this? Suppose my endpoint is used by a frontend and I want to display a proper message to user.
Catch the runtime exception at the fault boundary, and use the type and contents of the exception to craft an appropriate response to return to the client.
The fault boundary probably belongs out where the request is being handled, and you have awareness of things like the expected media type.
It wouldn't be completely unreasonable to have a fault boundary in your command handler, to package the DomainException as an ApplicationException, depending on how you feel about the domain leaking into the http layer. Since the validation of the command happens in the application component, you'll presumably want to be able to throw exceptions from there too. Both of these ApplicationExceptions would then be handled by the fault boundary in the component handling the http request.
And since it's not a checked exception, how should I handle it? Try/Catch for a runtime exception seems to be awful.
Embrace the truth that checked exceptions are an experiment that didn't really pan out; catch and report the problem.
DomainExceptions should be reported with 4xx status codes (ie: Client errors). 409-Conflict is the one I would prefer when the problem is a violation of the business invariant.
You are effectively telling the client that it can probably fix the problem (for example: get a fresh view of the aggregate state, and then modify the command so that it doesn't conflict). So it would be reasonable to include in your hypermedia response a link to a readable representation of the aggregate state, for example. Which would imply that the exception handling needs to happen in a place in your code that knows how to construct that link from the information in the request.
In pseudo code, the whole thing probably looks like
try {
// If this bit fails, we want to return a 400-Bad Request to the client.
Command<A> command = parse(httpRequest);
// We got this far, so now we pass the command we've parsed off to the
// command handler....
{
// The aggregate lookup could fail; that might be any of
// 400-Bad Request
// 404-Not Found
// 410-Gone
// ... depending on how you interpret the relationship between the
// *resource* and the *aggregate*
Aggregate<A> aggregate = Repository<A>.getById(command.id);
// A failure here is probably best reported via 409-Conflict;
// essentially to inform the user that the application state
// is stale, and should be refreshed
aggregate.execute(command);
// A failure here indicates that the command lost a data race;
// that's certainly a conflict. So if you are going to report
// an error at this point, 409-Conflict is the right way to go.
// A better choice might be to retry the command, though; after
// all, the stale version of the aggregate thought it was acceptable
// so the updated version might as well. In that case, you would
// catch this exception and reload the aggregate.
Repository<A>.save(aggregate);
}
// OK, we got through everything without a problem, so report success
reportSuccess(httpResponse);
} catch (...) {
// Possibly a single handler, or multiple handlers, depending upon
// how you design your exception hierarchy. The important thing is that
// the exception handler here knows how to translate the application
// exceptions thrown above into something suitable for http, and the
// media types being used by this endpoint.
exceptionHandler.forRequest(httpRequest).report(e, httpResponse);
}