Compiling and using LLVM Pass Plugin with icx / icpx

[Reuter_Jan]

Heya,

our team is interested in the development of an LLVM pass plugin for instrumentation purposes. While there already is the flag `-finstrument-functions`, there are reasons why we want to implement our own instrumentation.

Last week, I had a look at some of the LLVM based compilers and runtimes and checked if we would be able to compile and use a pass plugin. For Intel oneAPI, I saw that the required headers for building IR seem to be missing, but one could (in theory) enable a plugin, since the flag `-fpass-plugin=` is allowed and returns the error message `error: unable to load plugin '': 'Plugin entry point not found in ''. Is this a legacy plugin?'` when passing no path for the plugin.

I saw in an older post that generating IR seems to be supported with extra steps. 

My question is: How would one compile an LLVM pass plugin for Intel oneAPI? 

[NoorjahanSk_Intel]

Hi,

Thanks for posting in Intel Communities.

Could you please provide us with sample reproducer code and detailed steps you have followed so that we will investigate the issue at our end?

Also please provide us with Compiler version being used along with OS details.

Thanks & Regards,

Noorjahan

[Reuter_Jan]

Sure!

I'm using an example provided during a workshop I've attended a few weeks ago. This is the code, copied from the repository: 

#include "llvm/IR/Function.h"
#include "llvm/Pass.h"
#include "llvm/Passes/PassBuilder.h"
#include "llvm/Passes/PassPlugin.h"
#include "llvm/Support/CommandLine.h"
#include "llvm/Support/raw_ostream.h"
#include "llvm/Transforms/IPO/PassManagerBuilder.h"

using namespace llvm;

static cl::opt<bool> Wave("wave-goodbye", cl::init(false),
    cl::desc("wave good bye"));

namespace {

bool runBye(Function& F)
{
  if (Wave) {
    errs() << "Bye: ";
  } else {
    errs() << "Hello: ";
  }
  errs().write_escaped(F.getName()) << '\n';
  return false;
}

struct Bye : PassInfoMixin<Bye> {
  PreservedAnalyses run(Function& F, FunctionAnalysisManager&)
  {
    if (!runBye(F))
      return PreservedAnalyses::all();
    return PreservedAnalyses::none();
  }
};

} // namespace

llvm::PassPluginLibraryInfo getByePluginInfo()
{
  return { LLVM_PLUGIN_API_VERSION, "Bye", LLVM_VERSION_STRING,
    [](PassBuilder& PB) {
      PB.registerVectorizerStartEPCallback(
          [](llvm::FunctionPassManager& PM, OptimizationLevel Level) {
            PM.addPass(Bye());
          });
      PB.registerPipelineParsingCallback(
          [](StringRef Name, llvm::FunctionPassManager& PM,
              ArrayRef<llvm::PassBuilder::PipelineElement>) {
            if (Name == "goodbye") {
              PM.addPass(Bye());
              return true;
            }
            return false;
          });
    } };
}

#ifndef LLVM_BYE_LINK_INTO_TOOLS
extern "C" LLVM_ATTRIBUTE_WEAK ::llvm::PassPluginLibraryInfo
llvmGetPassPluginInfo()
{
  return getByePluginInfo();
}
#endif

Trying to compile the plugin with oneAPI yields the following results. The headers seem to be missing, but one could use the flag for plugins (minimal.c is a minimal compiling C program):

$ icpx --version
Intel(R) oneAPI DPC++/C++ Compiler 2023.1.0 (2023.1.0.20230320)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/software/software/Intel/2023.1.0/compiler/2023.1.0/linux/bin-llvm
Configuration file: /opt/software/software/Intel/2023.1.0/compiler/2023.1.0/linux/bin-llvm/../bin/icpx.cfg
$ icpx -fPIC -shared Bye.cpp
Bye.cpp:1:10: fatal error: 'llvm/IR/Function.h' file not found
#include "llvm/IR/Function.h"
         ^~~~~~~~~~~~~~~~~~~~
1 error generated.
$ icx -fpass-plugin= minimal.c                                                                                                           
error: unable to load plugin '': 'Plugin entry point not found in ''. Is this a legacy plugin?'

The same plugin compiles fine with LLVM 15.0.6 / 16.0.6 :

$ module load LLVM/15.0.6
$ clang --version
clang version 15.0.6 (https://github.com/llvm/llvm-project.git 088f33605d8a61ff519c580a71b1dd57d16a03f8)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/software/software/LLVM/15.0.6/bin
$ clang++ -fPIC -shared Bye.cpp -o Bye.so
$ clang -fpass-plugin=./Bye.so -O1 ./minimal.c
Hello: main

If I try to use the compiled plugin with Intel oneAPI, I get linking errors due to missing symbols, like llvm::EnableABIBreakingChecks and llvm::cl::GenericOptionValue.  

I've tested this with the following system.

- Pop_OS! 22.04

- Intel oneAPI 2023.1.0 (Base Kit + HPC Kit)

I was also able to reproduce the issue on several HPC systems, so this is not limited to my machine. 

[Reuter_Jan]

Are there any updates regarding the support for LLVM pass plugins?

[NoorjahanSk_Intel]

Hi,

We are also able to reproduce the issue at our end. We have reported this issue to the concerned development team. They are looking into your issue.

Thanks & Regards,

Noorjahan.

[NoorjahanSk_Intel]

Hi,

Thanks for your patience.

Intel icx/icpx compiler does not support LLVM Pass Plugin. 

We have provided this feedback to the relevant team. At this moment there is no visibility of when it will be implemented and available for use.

Please check the below Release notes for more details in future.

https://www.intel.com/content/www/us/en/developer/articles/system-requirements/intel-oneapi-dpcpp-system-requirements.html

Please let me know if we can go ahead and close this case.

Thanks & Regards,

Noorjahan.

[Reuter_Jan]

Heya Noorjahan,

thanks for the update. Its unfortunate to hear, that there's currently no support for LLVM pass plugins. For oneAPI, we will stick to the instrumentation option offered by -finstrument-functions-after-inlining including its drawbacks for tools like Score-P.

Hopefully the option to use pass plugins will be added soon.

You can go ahead and close the issue.

Kind regards,

Jan

[NoorjahanSk_Intel]

Hi,

Apologies for the inconvenience

The support for LLVM Pass plugin with right headers and library will be added in future releases.

>>You can go ahead and close the issue.

Thanks for the confirmation. We are going ahead and closing this case. We will no longer respond to this thread. If you require additional assistance from Intel, please start a new thread.

Thanks & Regards,

Noorjahan.

[NoorjahanSk_Intel · Accepted solution]

Hi,

We previously communicated that the feature you inquired about would be included in the upcoming release.

However, after careful consideration and evaluation by our development team, it has been decided that for now this feature wouldn't be a part of current product rodmap and hence won't be included in upcoming releases 

Apologies for any confusion or inconvenience this may have caused.

If you need any additional information, please post a new question. we would be happy to assist you

Thanks & Regards,

Noorjahan.

[Reuter_Jan]

Thanks for the update Noorjahan.

Its unfortunate to hear that it was decided to not include support for LLVM plugins in upcoming releases. For our use case, or more specifically when users use Intel compilers with Score-P, they will notice a significant higher overhead because we are not able to select or reduce the amount of instrumented functions during compile time like with GCC, LLVM and the AMD ROCm toolchain once the LLVM plugin is available. In addition, features in development might not work with Intel oneAPI compilers since they require having access to compiler plugins.

Kind regards
Jan

[NoorjahanSk_Intel]

Hi,

Apologies for the inconvenience. If you need any additional information, please submit a new question as this thread will no longer be monitored.

Thanks & Regards,

Noorjahan.