When using Compute Unit Replication, each compute unit will have its own set of memory ports. This results in a high amount of contention on the memory bus and if there are too many ports going to the memory interface, then it is very much possible that performance will start degrading.
