You can refer to the following PCIe avalong ST user guide, page 52 to understand the Channel placement and limitation.
The answer should be "You can assign other protocols to unused channels the if data rate and clock specification exactly match the PCIe configuration"
Thanks for your answer, I asked the question wrong.
I do not use the PCIe Hard IP Blocks, therefore the question is: If the speed of the transceiver channels next to unused PCIe Hard IP Blocks is limited? I'm afraid I can't use these channels as 5 Gbps transceivers.