This should work fine.
Depending on what you are actually doing, CPU and GPU might be competing for shared cache space, and even CPU time. Keep in mind that even your GPU workload might be demanding a bit of CPU time(for ex. the driver still has to perform a lot of operations on the CPU, again based on the type of work being sent)