Update: this issue is not specific to the cluster view, as I first noted. The error occurs for any grouping of VM's that number 39 or more. So if I'm in the VM tab of an esxi host that has 38 VM's, I can sort. 39+, and it throws the query service error.
In other news, I have been too busy/lazy to implement the certificate installations.