Question on xr and qr kr.

[Capchenxi]

Hello,

Thanks for your excellent work! I'm reading this paper and having one question on x^r and q_sa^r and k_sa^r. I'm wondering what is the specific relationship between q_sa^r ,k_sa^r and x^r? Since from the paper I learnt that q_sa^r and k_sa^r are calculated form the downsampled feature x_sa. And x_sa is basically has not impact on both x^t and x^r which means it is the same to x^t as well as x^r. I guess I didn't get the part of how you get q_sa^r ,k_sa^r from x^r. Would you please give a more detailed explanation or formula/figure on this? Thanks!

[deblinaml]

Hi, thanks for your interest! Yes, x is the same downsampled image representation from the encoder. For illustration purposes we wanted to show that x^t or x^r as the task steam 't' and reference task stream 'r', respectively. As far as q_sa^r and k_sa^r are concerned, they are the query and key from the encoder of the respective layers passed to the decoder of the respective layers, in the reference task stream. We then compute the attention as peer the formulation in the paper and propagate it to the other task streams. The propagated attention is then combined with the task-specific value of the task stream before passing to the multi head attention. Hope this clears things up.

[kejie-cn]

How do you select the reference task? Based on the paper, my understanding is that your choose surface normal estimation as reference empirically. Besides, I don't understand about this part, could you please explain it in more detail?