I've done something like that several years ago: x86 Decoder. I literally took a piece of squared paper, and in a 16x16 matrix representing each opcode I colored each square for opcodes that require a specific decoding. This revealed patterns that make it relatively easy to use binary logic to determine the decoding format of each instruction.
I believe my code is smaller than using tables, but not entirely optimal yet. Also, when translated to hardware each of the tests really just looks at a few bits so it could be quite compact.