Why register the relu operator in AutogradPrivateUse1 ?

[JingliangGao]

Hi,
In the pointwise_ops.cpp file, you register the relu operator in AutogradPrivateUse1, which is different with the relu_ operator. Why not register the relu operator in PrivateUse1 ? Just like the tan and tanh_ operators, which are registered in PrivateUse1.

[artyom-beilis]

Good question: dlprimitives provide several functions for activation so I tried to do something generic. After some time I reverted back to directly implementation but this remained.

So the answer is - historically, and probably it can be implemented as all other functions without a problem.

[JingliangGao]

In order to register the relu operator in PrivateUse1, I rewrite the function like that
`
Tensor & relu(Tensor & self)

{      
    GUARD;     
    dlprim::Tensor X = todp(self);     
    Tensor out = new_tensor_as(X.shape(),self);     
    dlprim::Tensor Y=todp(out);      
    dlprim::core::activation_forward(X,Y,dlprim::StandardActivations::relu,getExecutionContext(self));      
    sync_if_needed(self.device());      
    return self;     
}     

`
I have succeed to build this project, but failed in the running process as follows.

Could you provide more details about how to design a custom operator and then register it ? BTW, is it also possible to register the linear and max_pool2d operator in PrivateUse1?

[artyom-beilis]

As far as I remember max_pool2d and linear use custom ops because of slightly different input/output/gradient implementations for backpropogation by Torch operators and dlprimitives.

That is why custom backward function is applied. I used autograd private use any time I need little bit different backpropogation. I don't recall why I used it for ReLU but I assume I use it this way for max pooling because of different way I handle the location of "max" item for backpropogation.

I assume it can be rewritten differently but likely you'll need to change the kernels themselves in dlprimitives and run all tests.

Can you explain why it bothers you?

[JingliangGao]

Thank you for your patience, and I have understood what you mean. The reason why I want to do this is that —— inspired by the codegen method, which occurs in the op-plugin project https://github.com/Ascend/op-plugin/tree/master/codegen，so I prefer that all operators can be registered in the same dispatch key like"PrivateUse1" and then it would be very convient for me to write a codegen file to register all operators automatically at once. It seems like I have to write the codegen file carefully for different dispatch keys, and that does't mattter. Since I am not familiar with OpenCL, I do not currently have the ability to change the kernels themselves in dlprimitives.

[artyom-beilis]

Since I am not familiar with OpenCL, I do not currently have the ability to change the kernels themselves in dlprimitives.

Generally - not for every function you need to write kernels. Lots of operators are operations that can be implemented with basic broadcasting/reduction.

Take a loop on src/pointwise_ops.cpp - virtually none are implemented via just per element equation. And there is a huge amount of operators that can be covered with it. You mostly need to know the math for the operators. So the kernel is automatically generated from this expression and saves lots of headache.

But Some require very heavy work - convolutions, poolings, etc. matrix multiplications etc.

So there is lots of things that mostly require adding a function and testing it, for example GELU support change-set f2fff47

[artyom-beilis]

Tensor & relu(Tensor & self)

I assume you replacing aten::relu - it requires self to be const

Tensor relu(const Tensor & self); // {"schema": "aten::relu(Tensor self) -> Tensor", "dispatch": "True", "default": "False"}

[artyom-beilis]

Moved to discussions

Also note:

dlprim::Tensor X = todp(self);     

Would not work for strided tensors. Better method is this:

Tensor self_c = self.contiguous()
dlprim::Tensor X = todp(self_c);   

It would work on strided inputs

[JingliangGao]

Thinks for your timely help. I have solved this problem and it runs.