Possible to have FVP <-> actual device discrepancy?

#1

Hello,

We are attempting to run a lowered model on FVP (FVP_Corstone_SSE-300_Ethos-U55), but we have noticed that the numerics returned by the model on an actual device are not bit-accurate with the model running on FVP. We have successfully used this approach before and achieved bit-wise accurate reproduction of numerics, so we assumed it would always be the case. Are there any exceptions to the U55 simulation that we should be aware of?

Thanks,
Karan Dewan

#2

Hi Karan,

The FVP should be bit accurate and if it is not, it will count as a major bug. However, considering you are testing on the Arm Corstone-300 based FVP with Ethos-U55 I would say the likelihood for it to be a bug is low, as this platform has been tested quite extensively.

There are other factors which could explain differences too. Clock (or time) view of an FVP is not representative of real hardware, so race related bugs manifest differently on these systems. Numerical differences could be due to memory corruption if this was the case.

I’d recommend trying a couple of things:

  • If you haven’t already - run the same test on an MPS3 FPGA. Validate if you can get the same result as either the FVP or the hardware you’re running on.
  • Disable all CPU caches and observe if the problem is still there. I am not sure how much this will help, but may provide some clues.

I will pull in some other colleagues who may be able to help out better than I can.

Hope this is useful.
Kshitij

#3

Hi Karan,

It will also help to know:

  • Is the model executing completely on the NPU (no fall-back on CPU)?
  • Which ML framework is being used?
  • Are the compilation flags between targets exactly the same? Build logs may help here.
  • Is the “actual device” an FPGA implementation or existing hardware?

Thanks,
Kshitij