Adding ERF operator to TOSA

Hi All,

I wanted to propose/inquire about adding a unary elementwise operator : ERF – that computes gauss error function.

Ref : [link1]

It is commonly used in implementing a GELU operator (TF : tf.nn.gelu  |  TensorFlow v2.12.0 & PT : GELU — PyTorch 2.0 documentation).

The existing approaches uses a variety of approximations :

  1. polynomial based approximations
    a) CHLO->MHLO : [link2]
    b) torch-mlir : [link3]
  2. tanh based approximations (see PT link above)

However, depending on the requirement, they might not be accurate enough compared to an exact implementation.

Moreover, given the precedence of ops like tosa.tanh and tosa.sigmoid, would TOSA be open to adding tosa.erf ?

Here is a proposed defintion for a op in MLIR implementation :

//===----------------------------------------------------------------------===//
// Operator: erf
//===----------------------------------------------------------------------===//
def Tosa_ErfOp : Tosa_Op<"erf", [
    DeclareOpInterfaceMethods<InferShapedTypeOpInterface,
                              ["inferReturnTypeComponents"]>,
    Pure]> {
  let summary = "Computes gauss error function of input";

  let description = [{
    Gauss error function.
    For quantized integer data types, the TABLE operator should be used instead
    with the following definition.  The erf_table has 513 entries each of
    16-bit precision and covering the input range -4.0 to +4.0 in steps of 1/64.
  }];

  let arguments = (ins
    Tosa_Tensor:$input
  );

  let results = (outs
    Tosa_Tensor:$output
  );
}

(Working around the link restrictions)

[link1] : Error function - Wikipedia
[link2] : https://github.com/tensorflow/mlir-hlo/blob/4d922e8639edb6f03d5815bf33ca4ef17fd0202b/mhlo/transforms/chlo_legalize_to_hlo/chlo_legalize_to_hlo.cc#L592-L616

[link3] : https://github.com/llvm/torch-mlir/blob/100cb46baaf2338ae98694491d9a421a79eb3bd0/lib/Conversion/TorchToTosa/TorchToTosa.cpp#L2701-L2750

Hi Manupa,

Thanks for bringing this up. We are open to the possibility of adding ERF to the TOSA spec. As you point out, implementations are approximating ERF today. Do you have a sense of the accuracy you would want to see TOSA implementations require, and what implementation would be the reference source? Sigmoid requires exp(), but otherwise is regular floating point math.

As you note in the proposed MLIR op definition, we’d recommend TABLE implementations for 8 and 16 bit integer, as the TABLE values are up to the caller. There are no nonlinear pieces to ERF that would cause problems with the 16-bit interpolated table.

Thanks,
Eric

Hi Eric,

Thanks for the reply!

Short anwser is we’d like to be closer to std::erf implementation. When you say “Sigmoid requires exp()”, does that mean std::exp ? If thats acceptable, will it be possible to use std::erf as the reference?
From my quick skims, most runtimes end up using std::* in their references where possible.

Noted.

Finally, our goal is to be able to use tosa.erf in MLIR (which we are happy to contribute the lowerings to linalg as math.erf exist). However, would you be able to clarify the prerequisite steps that has to be done here in terms of contributing to the tosa spec ?

Yes, I think std::erf is a reasonable reference. My apologies as I forgot that it existed. My other concern was whether GPUs had ERF implemented. I can see a version exists in OpenCL C, that should indicate that GPUs could implement this in shader.

I agree that if we put this in the MLIR dialect, we should have a matching change in the TOSA spec. We have two choices. We would welcome you writing up a patch to the spec and uploading it to https://review.mlplatform.org in the tosa/specification repository. There is a contribution agreement it would ask you to sign. Once you’ve uploaded it, we would do a review and then merge it into the specification.
Alternately, I can treat this like a feature request, and can write up the spec change, posting it to gerrit for you to review.

I’m happy with either option, let me know which you prefer.

Eric

Hi Eric,

Apologies for the delay!

My other concern was whether GPUs had ERF implemented. I can see a version exists in OpenCL C, that should indicate that GPUs could implement this in shader.

Yes, this is generally available.
[opencl link]
https://docs.nvidia.com/cuda/cuda-math-api/group__CUDA__MATH__SINGLE.html
https://docs.amd.com/bundle/HIP-API-Guide-v5.4/page/a00161_source.html

Alternately, I can treat this like a feature request, and can write up the spec change, posting it to gerrit for you to review.

Thanks for the offer! It feels it might be efficient for me to review rather than learning the process, if that is something you are happy to do. As I commited before, we will do the MLIR side of things. What do you think ?

[opencl link] : erf, erfc

Sounds good to me. I’ll write up the spec change and post a link. I should have it by early next week.

I haven’t forgotten this, I’m going through internal checks and should be able to post for review tomorrow.

No worries!

cross posting the MLIR review : ⚙ D150357 [MLIR][TOSA] add tosa erf operator

Here is the review for the spec change: https://review.mlplatform.org/c/tosa/specification/+/9622

I’ll take a look at the MLIR review.