Replacing an existing KWS model with a new one

Hi,

I have been trying to replace an existing KWS model in the following repo, although it is a private repo, but you might have access to that:
https://github.com/alifsemi/ml-kws-ic-example-ensemble-share
Firstly, I was wondering if the documents are up to date.
Because I found that the structure of directories/subdirectories have changed a bit but seemingly it is not updated in the document.
Aside from that, in KWS.md document, it has been said that for replacing micronet_m model with a different model (let’s say a previous one such as dnn-m-quantized), we just need to point to the .tflite file, however, I tracked the commits that your team pushed and found out that there are some files (.cc and header files) in the source directory, have been modified for replacing the DS-CNN with Micronet-m.
That’s why I am a bit confused if either the document is updated or we need to modify different files in order to be able to change the model.

Any help would be greatly appreciated
Thank you in advance,
Sina

Hi Sina,

When changing to a different model for one of the usecases, e.g. KWS, you will need to point to the different TFLite file when building the application. Also, as you have worked out already, will need to make changes to the .hpp and .cc file that abstracts details about the model. For KWS this will be MicroNetKwsModel.hpp and MicroNetKwsModel.cc

This only needs to be done if the operators used by the new TFLite model are different to those used by the existing model. If they are then you will need to add these to the EnlistOperations method (source/application/api/use_case/kws/src/MicroNetKwsModel.cc - ml/ethos-u/ml-embedded-evaluation-kit - Gitiles) and also change this value here: source/application/api/use_case/kws/include/MicroNetKwsModel.hpp - ml/ethos-u/ml-embedded-evaluation-kit - Gitiles if more operators are now in the model.

Please be aware that the repo you have linked to is likely from a very old fork of our ml-embedded-evaluation-kit repo so may be a bit out of date now. I will also check if documentation in our repo can be clearer about this change that needs to be made when swapping model.

Let me know if any of this is unclear or you need some more assistance and we will be happy to help.

Many thanks,
Richard

Hi Richard,

First, I appreciate your time giving to that for replying to my concern.
Based on the document, the mentioned flow is exactly what I understood. I must modify the enlisted operations in accordance with the operations in a model in this case a KWS model and the number of max operations also need to adapt.
However, for my information (since I need to summarize a full replacement flow), there are two possible cases that should be considered:
1- Replacing the model with a different model in the same repository (ML Zoo)
2- Replacing with a completely different model (from other external repository, not ML Zoo)

For the first part, let’s think of replacing the micornet model with dnn medium quantized (although this is an old version of ARM KWS model, for our consideration, it might be helpful). I changed the enlisted operation to:

bool arm::app::MicroNetKwsModel::EnlistOperations()
{
    //this->m_opResolver.AddReshape();
    //this->m_opResolver.AddAveragePool2D();
    //this->m_opResolver.AddConv2D();
    //this->m_opResolver.AddDepthwiseConv2D();
    this->m_opResolver.AddFullyConnected();
    this->m_opResolver.AddRelu();
    this->m_opResolver.AddSoftmax();
    this->m_opResolver.AddQuantize();
    this->m_opResolver.AddDequantize();
    if (kTfLiteOk == this->m_opResolver.AddEthosU()) {
        info("Added %s support to op resolver\n",
            tflite::GetString_ETHOSU());
    } else {
        printf_err("Failed to add Arm NPU support to op resolver.");
        return false;
    }
    return true;
}

Then, I changed the max operation, but the indexes of input and output also has changed to this (since the input shape is (1,250)), please let me know if this modification looks good and these places are the only parts that should be modifed:

For the second part, if the .tflite model is not implemented by ARM, what are the other files that should be modifed, I am thinking of a different MFCC, and other factors.

Please accept my apologies for providing and asing for too much detail here.
Any comment and feedback would be greatly appreciated.

Best,
Sina

Hi Sina,

Thanks for the questions raised. You’re right in thinking that there might be other changes required especially if the model you’re looking at has custom preprocessing steps (like you say MFCC - if the model has been trained with a different pipeline, this would be different). From a high level perspective you would have to look at:

  • Model ops (covered in Richard’s post and our documentation) enlisted.
  • Row/col indices in the model header file (like the one you pointed to in your response).
  • Preprocessing steps
    • MFCC for example - any custom way of calculating the coefficients that are fed in.
  • Post processing steps
    • Labels (the new model might have different keywords or their indices might be different)

For the models in the Arm model zoo, we’ve had to deal only with ops, indices and labels. You might find this CR link useful. It covers the migration from the old model to the current micronet one.

Useful changes:

Hope this helps.

Kshitij

1 Like

One thing to add to Kshitij’s response:

If switching to the other Arm trained networks such as DNN medium the preprocessing (MFCC feature extraction) is basically the same so you shouldn’t need to change this. However, you will need to adjust the feature FrameLength and FrameStride values set here: usecase.cmake

The DNN variant of KWS models I believe should use values of 640 for both FrameLength and FrameStride, while others used 320 for the stride. Changing to 640 for both should then change the size of the final features to match the input shape of (1, 250) for that model.

1 Like

Kshitij and Richard,

Thank you so much for these information.
I will try to put them into effect to see if I could get the desired output.
Once again appreciate your help and please let me continue this discussion if I come up with any other issue.

My best regards,
Sina