TOSA Reference model test execution fail

TOSA
#1

Hello,
I noticed that when running some of the AvgPool2d conformance tests the reference compliance model fails if debug logs are enabled.

$ tosa_verif_run_tests --ref-model-path build/reference_model/tosa_reference_model --sut-module runner.tosa_refmodel_sut_run -t conformance/operators/tensor/avg_pool2d/avg_pool2d_1x2x65533x2_f32_accf32_st11_kern22_pad1000_s0 
Running 1 tests on 1 system under test
runner.tosa_refmodel_compliance_sut_run: Result code PASS avg_pool2d_1x2x65533x2_f32_accf32_st11_kern22_pad1000_s0
runner.tosa_refmodel_sut_run: Result PASS avg_pool2d_1x2x65533x2_f32_accf32_st11_kern22_pad1000_s0
Totals (runner.tosa_refmodel_sut_run): 1 expected_pass, 0 expected_failure, 0 unexpected_pass, 0 unexpected_failure, 0 internal_error, 0 skipped
Totals (runner.tosa_refmodel_compliance_sut_run): 1 expected_pass, 0 expected_failure, 0 unexpected_pass, 0 unexpected_failure, 0 internal_error, 0 skipped

$ tosa_verif_run_tests --ref-model-path build/reference_model/tosa_reference_model --sut-module runner.tosa_refmodel_sut_run -t conformance/operators/tensor/avg_pool2d/avg_pool2d_1x2x65533x2_f32_accf32_st11_kern22_pad1000_s0 --ref-debug low 
Running 1 tests on 1 system under test
'utf-8' codec can't decode byte 0xf3 in position 1206: invalid continuation byte
runner.tosa_refmodel_compliance_sut_run: Result code FAIL avg_pool2d_1x2x65533x2_f32_accf32_st11_kern22_pad1000_s0
runner.tosa_refmodel_sut_run: Result PASS avg_pool2d_1x2x65533x2_f32_accf32_st11_kern22_pad1000_s0
UNEXPECTED_FAILURE conformance/operators/tensor/avg_pool2d/avg_pool2d_1x2x65533x2_f32_accf32_st11_kern22_pad1000_s0 (runner.tosa_refmodel_compliance_sut_run)
Totals (runner.tosa_refmodel_sut_run): 1 expected_pass, 0 expected_failure, 0 unexpected_pass, 0 unexpected_failure, 0 internal_error, 0 skipped
Totals (runner.tosa_refmodel_compliance_sut_run): 0 expected_pass, 0 expected_failure, 0 unexpected_pass, 1 unexpected_failure, 0 internal_error, 0 skipped

Using Tosa reference model main branch(da0249d5). Tests generated with:

$ tosa_verif_conformance_generator \
  --profile tosa-pro-fp          \
  --output-directory tosa_reference_model/conformance \
  --ref-model-path tosa_reference_model/bin/tosa_reference_model \
  --flatc-path tosa_reference_model/bin/flatc \
  --schema-path tosa_reference_model/tosa.fbs \
  --generate-lib-path tosa_reference_model/lib/libtosa_reference_generate_lib.so
#2

We’ll have to investigate this, at first glance it looks like a bug in the reference model. I can reproduce it locally, will let you know when we have more information.

1 Like
#3

Yes, this was a bug in the reference model, now fixed with this change: https://review.mlplatform.org/c/tosa/reference_model/+/13742

The debug log code could read off of the end of an array.