Any restrictions on custom models?

ML Embedded Evaluation Kit
#1

Hey Guys,

I have been trying out ml-embedded kit and getting used to it. I tried to deploy a custom model on MPS3 board and faced a following error.

INFO - V2M-MPS3 revision C
INFO - Application Note AN552, Revision B
INFO - MPS3 build 2
INFO - MPS3 core clock has been set to: 32000000Hz
INFO - CPU ID: 0x411fd220
INFO - CPU: Cortex-M55 r1p0
INFO - Arm Corstone-300 - AN552 platform initialised
INFO - ARM ML Embedded Evaluation Kit for MPS3 FPGA and FastModel
INFO - Target system design: Arm Corstone-300 - AN552
INFO - Version 21.11 Build date: Jan 26 2022 @ 14:34:51
INFO - Copyright (C) ARM Ltd 2021. All rights reserved.
INFO - Creating allocator using tensor arena in SRAM
INFO - Allocating tensors
Didn't find op for builtin opcode 'PAD' version '2'. An older version of this builtin might be supported. Are you using an old TFLite binary with a newer model?

Failed to get registration from op code PAD

[ERROR] allocateTensors() failed
ERROR - tensor allocation failed!
ERROR - Failed to initialise model
INFO - program terminating..

I took a tf.keras model and converted it into a tflite model by the following steps.

converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.representative_dataset = rep_dataset
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8
tflite_model = converter.convert()

It would be helpful if I could get some insights if possible.
I’m using TF 2.5.0 to convert TF Keras model into TFLite.

#2

Hi Surya,

We are sorry for a late response.

Answering your general question about models restrictions:

There are some restrictions on models that could be inferenced and they relate to operators support on different back-ends. Models that are to be optimized to run on NPU should comply with these requirements: Supported Ops

Models that to be executed with CPU should have operators supported by Tensorflow Lite Micro framework. You can check micro operators resolver registration functions to see what’s available: https://github.com/tensorflow/tflite-micro/blob/main/tensorflow/lite/micro/micro_mutable_op_resolver.h

Your particular error message “Didn't find op for builtin opcode ” suggests that micro op resolver does not know about PAD operator that your model has.

If you are using tflite::AllOpsResolver then it could be a matter of Tensorflow and Tensorflow Lite Micro versions mismatch, try using newer Tensorflow version. For example, we are successfully converting to tflite models with version 2.7.

If you are not using AllOpsResolver but manually register ops for your model, please, make sure that all ops from your model were listed. As we did it here: source/use_case/vww/src/VisualWakeWordModel.cc - ml/ethos-u/ml-embedded-evaluation-kit - Gitiles

I hope this helps. Please, let us know if you are able to resolve the problem.

#3

Hey

Sorry for the delayed reply. I couldn’t try out the suggested solution earlier. Thanks a lot for your pointers regarding supported TFLite operators.

I was able to fix Didn't find op for builtin opcode the error by manually registering the PAD operator as you suggested.

Could you show me a example of how to use tflite::AllOpsResolver so that I don’t have to manually register all ops like this this->m_opResolver.AddPad();? My use case is img_class (image classification).

#4

It could be that you need to add PadV2 instead of just Pad for your use case, so you could try:
this->m_opResolver.AddPadV2(); instead.

For how to use the AllOpsResolver you can check the code in the inference_runner use case. Specifically in TestModel.hpp/cc and the code relating to tflite::AllOpsResolver.

Note that using the AllOpsResolver will use a lot more memory than registering only the operations you need, so if memory is an issue it might be better to stick with the current MicroMutableOpResolver and manually registering the ops required.

#5

Thanks for the suggestions! I’ll look more into AllOpsResolver.

I actually ran into another issue too.

Didn't find op for builtin opcode 'MEAN' version '2'. An older version of this builtin might be supported. Are you using an old TFLite binary with a newer model?

Failed to get registration from op code MEAN

My EnlistOperations looks like this

bool arm::app::MobileNetModel::EnlistOperations()
{
    this->m_opResolver.AddDepthwiseConv2D();
    this->m_opResolver.AddConv2D();
    this->m_opResolver.AddAveragePool2D();
    this->m_opResolver.AddAdd();
    this->m_opResolver.AddReshape();
    this->m_opResolver.AddSoftmax();
    this->m_opResolver.AddPad();
    this->m_opResolver.AddMean();

Any idea why I’m still getting the same error for MEAN?

#6

I was able to resolve this. Apparently there’s variable to track the maximum number of Ops that can be enlisted in include files.

Pasting the code for future references.

   class MobileNetModel : public Model {

    public:
        /* Indices for the expected model - based on input tensor shape */
        static constexpr uint32_t ms_inputRowsIdx     = 1;
        static constexpr uint32_t ms_inputColsIdx     = 2;
        static constexpr uint32_t ms_inputChannelsIdx = 3;

    protected:
        /** @brief   Gets the reference to op resolver interface class. */
        const tflite::MicroOpResolver& GetOpResolver() override;

        /** @brief   Adds operations to the op resolver instance. */
        bool EnlistOperations() override;

        const uint8_t* ModelPointer() override;

        size_t ModelSize() override;

    private:
        /* Maximum number of individual operations that can be enlisted. */
        static constexpr int ms_maxOpCnt = 10;

        /* A mutable op resolver instance. */
        tflite::MicroMutableOpResolver<ms_maxOpCnt> m_opResolver;
    };