EncoderDecoderFactory - Questions

[Atmoboran]

Hello everybody,
so I'm currently playing around with the EncoderDecoderFactory and got some questions:

build_model
In build_model it says:
task (str) – Task to be performed. Currently supports "segmentation" and "regression".
-> How can I build a model for a classification task?
There is in general no argument to give a head as an argument, how do I do this? Or does the 'task' define automatically which head is chosen?

registries
Is there a documentation of all the registries, i.e. can I for example look up somewhere (except the source code) what a certain decoder does and to what encoder and head it is compatible to? How can I find out which combinations of encoder-neck-decoder-head are possible? How do I know if a decoder needs a head or not?

And a general comment:
I see that there are many example.yaml files, but for me they are much less helpful than a jupyter notebook, which shows step by step workflows and results!

I hope these are not too stupid questions, as I said before, I'm not a computer scientist, but I hope these questions are valuable for you to understand what non-AI-expert users struggle with.

Best regards,

Boran

[Joao-L-S-Almeida]

Hi, @Atmoboran
buil_model
I believe it's an error in the class annotation, since it should work for all the tasks. Check the EuroSat example: terratorch/examples/confs/eurosat.yaml. Let me know if this example is working for you. I'll update the docs about it.

registries
You can get more information about the registries in this page of the documentation: https://ibm.github.io/terratorch/registry/
In fact, a complete guide describing combinations between encoders, necks and decoders is still missing, although we are extending the documentation to provide more detailed information.
Generally speaking, you can basically combine any kind of encoder e decoder, you just need to use the appropriate neck to adjust outputs and inputs. For example, by using the neck PermuteDims when the decoder is expecting a different order for the dimensions outputted by the encoder.
You usually don't need to care about the heads, since they are handled by the tasks. For instance, in terratorch/tasks/pixel_wise_model.py:143:

    def _get_head(self, task: str, input_embed_dim: int, head_kwargs):
        if task == "segmentation":
            if "num_classes" not in head_kwargs:
                msg = "num_classes must be defined for segmentation task"
                raise Exception(msg)
            return SegmentationHead(input_embed_dim, **head_kwargs)
        if task == "regression":
            return RegressionHead(input_embed_dim, **head_kwargs)
        msg = "Task must be one of segmentation or regression."
        raise Exception(msg)

General comments
I totally agree with you. Although the yaml files are the most practical way to run these experiments via CLI, the notebooks are more illustrative and can preserve results that, mostly part of the times, the user cannot replicate due to limitations in computational resources. We will try to extend it.
Thanks for you observations.

[Atmoboran]

Hi @Joao-L-S-Almeida ,
thanks for the answers.
Yes I realized the task = 'classification' is working, so only the documentation needs an update.

Ok, so do I understand correctly that there is only one Head per possible task instead of a variety of heads per task?

A documentation for the combinations would indeed be nice to know which combination is appropriate for someones task. I think when various options are possible, it is always good to have some information so the users can make an informed decision, but I understand that this is probably something for long-term, when there are more examples and experiences.

Best regards,
atmoboran

[Joao-L-S-Almeida]

About the heads, unless for our current approach, it is basically that, @Atmoboran. You still can pass arguments to the head, but the architecture is kept as immutable.
This is something that must be made more flexible for the next releases.