The programmer sees both the FPGA core and the HPS.
I saw that. It's not related. I got it working though. If anyone else runs into this it mat help.
First off, the problem was that when I tried to browse for the connection in debug configurations, it gave a python error with no information. I expected the connection to show up or be able to browse for it.
As a hail mary, I pasted the USB hardware ID into the connection box and tried that. It failed of course, but once I did that, when I hit the browse button, the connection showed up and worked fine. I doubt it had anything to do with the hardware ID, maybe just having some text there so I could attempt to debug. Anyway, I can't reproduce the problem because the USB blaster shows up now.
Maybe this will help someone else.