I am trying to implement an autoencoder architecture on Tensorflow.
Specifically, the tensor
Zis a compressed representation of the original input, with size (1, 1, 1, 192). I am trying to apply a deconvolution layer for multiple times to scale back up to the input size (1, 1024, 32, 1).
The first deconvolution layer would be this:
def Decoder_test_same(self, Z): with tf.variable_scope("Decoder") as scope: conv1 = tf.layers.conv2d_transpose(Z, 192, 3, strides=2, padding='same', activation=tf.nn.relu) return conv1
which uses the 'SAME' padding so the dimension of
conv1 will be (1, 2, 2, 192). This is followed by another 8 deconvolution layers with different kernel sizes / no. of channels.
Unfortunately it seems that NCSDK does not support 'SAME' padding:
[Error 5] Toolkit Error: Stage Details Not Supported: Wrong deconvolution output shape.
After looking up for solutions on the internet I decided to use the 'VALID' padding instead. This should give me a tensor with size (1, 3, 3, 192). By removing the last row and column, I should get the exact same (1, 2, 2 ,192) tensor as previously mentioned.
The new code now looks like this:
def Decoder_test_valid(self, Z): with tf.variable_scope("Decoder") as scope: conv1 = tf.layers.conv2d_transpose(Z, 192, 3, strides=2, padding='valid', activation=tf.nn.relu) conv2 = tf.slice(conv1, [0, 0, 0, 0], [1, 2, 2, 192]) return conv2
mvNCProfile can now handle the deconvolution layer, but it now complains that the Slice operation is not supported:
[Error 5] Toolkit Error: Stage Details Not Supported: Slice type not supported
tf.slice op completely unsupported, or are there some restrictions on the input that I failed to observe? If slicing is indeed not supported, is there some alternative where I can obtain the subset of a tensor? I have also tried
tf.split but that resulted in Error 5 as well.
Thanks in advance
@ferrophile The current NCSDK (v 2.08.1) doesn't have support for slicing a tensor dimension like how you showed. The NCSDK currently requires index 1 and index 2 from the input tensor and sliced tensor to be equal.
@Tome_at_Intel Thanks for the response. I realized that even if I got past the slicing problem, my network still wouldn't compile since I use batch norm layers after every deconvolution layer, which results in Error 5 again:
[Error 5] Toolkit Error: Stage Details Not Supported: FusedBatchNorm must be preceded by convolution or fully connected layer
Can I know if there is some official documentation on what rules I have to follow when using a particular layer in Tensorflow? I think I saw a list of supported layers somewhere in the forum, but apparently there are certain limitations on the layer arguments, dimensions etc.
By the way, I also tried to compute the batch normalization process manually using stored parameters. NCSDK reported that it does not support the
var = tf.get_variable('batch_normalization/moving_variance', initializer=tf.zeros(192, tf.float32)) std = tf.sqrt(var + 0.001)
[Error 5] Toolkit Error: Stage Details Not Supported: Sqrt
tf.sqrt completely unsupported, or are there specific requirements regarding the input tensor?
@ferrophile There are still some TensorFlow ops that the NCSDK doesn't have support for yet and it seems like we don't have support for the sqrt operation at the moment.
Thanks. I am trying to find a workaround solution using the ops supported right now, so I had a look at the release notes of the current version:
For V1.12.01 of the SDK, "Crop" is a firmware feature according to the release notes. Could you tell me which Tensorflow op could I use to make use of this feature? (assuming this feature is not for Caffe only)