using multiple sawtooth validators with SGX mode

[Nishida__Yoshi]

Hello,

I am wondering how to run multiple validators with SGX mode. If I use simulation mode, I can run multiple validators without problems.

However, when I use SGX mode, I can only launch the first validator successfully. I got the error message below from other validators. As a result, these validators cannot sign blocks. (Hence, we will ran into z-test failure eventually)

It seems that we can solve this issue if we can get another SPID, however it seems that we can get more than 1 SPID per a customer.

If someone knows how to avoid this issue, please let me know.

Thanks,

018-04-30 13:50:31.005 CRITICAL publisher] on_chain_updated exception.
[2018-04-30 13:50:31.005 ERROR    publisher] Failed to create monotonic counter.: SGX ERROR: SGX_ERROR_MC_OVER_QUOTA
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sawtooth_validator/journal/publisher.py", line 743, in on_chain_updated
    self._build_candidate_block(chain_head)
  File "/usr/lib/python3/dist-packages/sawtooth_validator/journal/publisher.py", line 602, in _build_candidate_block
    if not consensus.initialize_block(block_builder.block_header):
  File "/usr/lib/python3/dist-packages/sawtooth_poet/poet_consensus/poet_block_publisher.py", line 301, in initialize_block
    poet_enclave_module=poet_enclave_module)
  File "/usr/lib/python3/dist-packages/sawtooth_poet/poet_consensus/poet_block_publisher.py", line 131, in _register_signup_information
    nonce=nonce)
  File "/usr/lib/python3/dist-packages/sawtooth_poet/poet_consensus/signup_info.py", line 85, in create_signup_info
    nonce[-cls.__MAXIMUM_NONCE_LENGTH__:])
  File "/usr/lib/python3/dist-packages/sawtooth_poet_sgx/poet_enclave_sgx/poet_enclave.py", line 660, in create_signup_info
    signup_data = _create_signup_data(originator_public_key_hash)
SystemError: Failed to create monotonic counter.: SGX ERROR: SGX_ERROR_MC_OVER_QUOTA

[Michael_S_Intel]

Hi Yoshi,

This is the expected behavior and part of the PoET security design: you should be able to run at most one PoET enclave per SGX node ..

-michael-

[Nishida__Yoshi]

Hi Michael,

Thanks for the response. Is there a way to run PoET enclave on each SGX node with one SPID?

I have requested  4 SPIDs as I have 4 validator nodes. However, I only got one SPID and other requests were declined.

--

Yoshi

[Hoang_N_Intel]

Because the error also indicates "SGX_ERROR_MC_OVER_QUOTA ," the guidance at https://software.intel.com/en-us/node/709159 stated that "To avoid exhausting the available quota, an Intel® SGX application should record the MC UUID that sgx_create_monotonic_counter_ex returns and destroy a MC when it is not needed any more."

As a test, can you help reinstall the SGX PSW as one way to reset the counter and run it again to see whether you can get past that error or not? Please let us know the test result.

[Nishida__Yoshi]

Hi Nguyen,

Thanks for the response. I've uninstalled SGX software (by running /opt/intel/sgxpsw/uninstall.sh) and reinstall it on all my validator nodes.

However, it seems that I got the same SGX_ERROR_MC_OVER_QUOTA error..

BTW, I just followed the instructions on the following URL to install the required software. Is it enough?

https://sawtooth.hyperledger.org/docs/core/releases/1.0.1/sysadmin_guide/configure_sgx.html

[Dan_Middleton]

Hi,

Yes PoET with SGX is designed to run one instance per machine. The intent is something like the Satoshi vision for Proof of Work: one computer, one vote. PoET simulator is designed to let you experiment with the protocol more easily and so you can deploy several nodes on the same physical host. 

The current design of PoET uses a SPID like an identifier for the network, so you don't really want to get multiple SPIDs. If you want to test with multiple Sawtooth nodes in SGX you'll need to have separate hosts. (Or again you can build test networks using PoET with the simulated enclave and use several instances on the same host)

The PoET SGX code could do a better job cleaning up monotonic counters. If it gets stuck like that the easiest way to clean all of the counters is to reinstall Platform Services (PSW). That should reset the counters. Perhaps try restarting the machine and reinstalling PSW again. 

Regards,
Dan

[Hoang_N_Intel]

When you uninstall the SGX PSW AESM service, please make sure that you remove the installed directory completely (e.g  /var/opt/aesmd) before you re-install it. Please give it another try and see whether it helps or not. Thanks.

[Nishida__Yoshi]

Hi Don, Hoang,

Thanks for the response. After I remove /var/opt/aesmd and re-installed driver and PSW, it seems it starts working properly.

Thanks for the help.

--

Yoshi