thepossible reason of the failure you described is that VTune 8.0 doesn't support working on Vista. Vista compatibility is declared starting 9.0, so you'd better to try your luck with the lates version that is 9.0 Update 7.
The other actions you did look correct.
Here, on forum, you can get help on quite common (known) issues or ones that simple to investigate.
Sorry, but your problem looksspecific and needs to be investigated deeply in official way.
Could you submit your issue on https://premier.intel.com?
What is the point of instrumenting the plugin inside the Photoshop? I understand that it would be the most relevant and easiest way but I believe there is no need for that.
Why don't you just profile your processing code outside of the plugin? Make a test application and copy the main processing function into it and then simulate some real workload. Once you manage to tune the function to the desired speed just merge the changes into the plugin itself.
At least that is how I would do it if I had the same problem.