MACE of provided model

[dkoguciuk]

Hi @ruizengalways ,

What MACE should the provided model have? I've downloaded pfnet_0200.h5 , run the eval script and ended up with:
MACE Metric: 7.451346

It sounds like a way worse quality than reported in the paper. Could you please help me to reproduce your results?

Best,
Danie;

[ruizengalways]

Did you test this model in COCO 2014 dataset? Since the keras and tensorflow version used in this code repo are too old, some papers reproduced this paper and achieved better results than that in this paper. (https://arxiv.org/abs/2004.02132) The network architecture is quite easy and the novelty is perspective field and RANSAC, it is not that hard to be reproduce it using modern deep learning framework such as pytorch and tensorflow2.
Best regards,
Rui

[dkoguciuk]

Hi Rui,

thanks for the replay.

1. I am in the process of reproducing your results and I got lower numbers than reported. That's why I've tried to use your repo.
2. And yes, I did test it using your model and your code. What MACE should the provided model achieve?
3. I've seen that a couple of settings in your code does not match the settings described in the paper, which one should I believe more? :)

Best,
Daniel

[ruizengalways]

Hi Rui,

thanks for the replay.

1. I am in the process of reproducing your results and I got lower numbers than reported. That's why I've tried to use your repo.
2. And yes, I did test it using your model and your code. What MACE should the provided model achieve?
3. I've seen that a couple of settings in your code does not match the settings described in the paper, which one should I believe more? :)

Best,
Daniel

What deep learning framework did you use to reproduce this paper. If you use pytorch, please be careful with the initialization method, which is totally different from that in the tensorflow. Also, please lower learning rate to 0.0001. and batch size to 32. Did you use

PFNet/evaluation.py

Line 318 in 4aa36e6

print("MACE Metric: ", final_mace)

this to obeserve results?

PFNet/model.py

Line 66 in 0b000ab

def mace(y_true, y_pred):

This mace function is used to monitor the training process rather than the final metric because it does not use RANSAC.

I will be refurbishing and organize this repo using pytorch. Thanks for your attention

Best regards,
Rui

[dkoguciuk]

Hi Rui,

let's forget my implementation at the moment, it does not matter from the perspective of my questions. Now I am using YOUR code and YOUR provided model. I evaluate the model using YOUR script:

python evaluation.py --dataset=/data/input/datasets/COCO --model=pfnet_0200_orig.h

And it gave me the MACE I've previously mentioned: MACE Metric: 7.451346. Do you have a clue what's going on? You can reproduce it by just cloning your repo as it is and running the evaluation script by yourself. Something is wrong here, and I would love to understand what :)

Best,
Daniel

[ruizengalways]

Hi Rui,

let's forget my implementation at the moment, it does not matter from the perspective of my questions. Now I am using YOUR code and YOUR provided model. I evaluate the model using YOUR script:

python evaluation.py --dataset=/data/input/datasets/COCO --model=pfnet_0200_orig.h

And it gave me the MACE I've previously mentioned: MACE Metric: 7.451346. Do you have a clue what's going on? You can reproduce it by just cloning your repo as it is and running the evaluation script by yourself. Something is wrong here, and I would love to understand what :)

Best,
Daniel

Hi Daniel,

I clearly remember that this network can achieve around 3.2 MACE without any further hyperparameters tuning. The results shown in the paper is obtained after fine-tuning. MACE 7~ is little bit too low. I will investigate it. So far, I do not have CUDA 9 and COCO dataset to run this code again. I will install them and test this program see how it goes.

Best regards,
Rui

[dkoguciuk]

Hi Rui,

sure, thank you!

BTW. could you please elaborate a bit about this MACE? You said it can achieve 3.2 MACE, but in the paper, you've reported 1.63 MACE - so where the difference comes from?

Best,
Daniel

[ruizengalways]

Hi Daniel,

The 1.63 is achieved after fine-tuning hyperparameters. In my initial experiements, I used a lot of default settings, such as Adam optimizer, l2 loss, etc., this network can achieved ~3 MACE after 40 epochs.

Best regards.
Rui

[dkoguciuk]

Oh, I see. Do you remember / did you write down the hyperparameters that gives MACE smaller than 2.0?

[ruizengalways]

As mentioned in the supplementary, https://static-content.springer.com/esm/chp%3A10.1007%2F978-3-030-20876-9_36/MediaObjects/484523_1_En_36_MOESM1_ESM.pdf , the results obtained from smooth l1 loss is always better than that of l2 loss. Also, training this network longer can always improve the performance. I only reported 200 epoches results in this paper since I needed to catch the deadline of the conference submission. I did not have enough time to train this network longer but I believe that higher performance can be obtained if train it longer. The hyperparameters is not very hard to find. Please try different batch sizes (16, 32, 64). You can also try learning rate (0.001 or 0.0001) and divide it by 10 after each 100 epoches. I recommend you to reimplement it using pytorch because I also met some problems with it one year ago. I installed keras and tf but the weights cannot be updated after each iteration. I guess this is caused by the wrong version of the softwares.

Best regards,
Rui

[dkoguciuk]

Hi Rui,

thank you for all the tips and discussions. I've managed to reproduce your results with my implementation - I got 1.47 MACE on the COCO test with 5.7M of training images (the same number as in DeTone paper).

Now, I am thinking about the correctness of your comparison with DeTone (in the Table 1 of your paper). You're using ResNet50 encoder + ResNet50-like decoder and you're comparing yourself with VGG-like encoder proposed by DeTone - it's not a fair comparison in my opinion. VGG is an old backbone without many modern tricks (i.e. batch-norm layers) and especially without residual connections, which in case of homography estimation could play an essential role. I've implemented the DeTone method with ResNet34 backbone and got 1.89 test MACE, which is a WAY better than 9.20 reported in the paper with HomographyNet.

What's interesting, in the paper you haven't emphasized enough the best feature of your method in my opinion: outlier robustness. By using RANSAC in the inference phase this method can be robust to outliers / foreground / dynamic objects. And it was confirmed by Le in their Deep Homography Estimation for Dynamic Scenes paper.

Thank you one more time for the discussion.

P.S. I won't close the issue, since the problem is not solved so it can help anyone interested in using your repo.

Best,
Daniel

[ruizengalways]

Hi Rui,

thank you for all the tips and discussions. I've managed to reproduce your results with my implementation - I got 1.47 MACE on the COCO test with 5.7M of training images (the same number as in DeTone paper).

Now, I am thinking about the correctness of your comparison with DeTone (in the Table 1 of your paper). You're using ResNet50 encoder + ResNet50-like decoder and you're comparing yourself with VGG-like encoder proposed by DeTone - it's not a fair comparison in my opinion. VGG is an old backbone without many modern tricks (i.e. batch-norm layers) and especially without residual connections, which in case of homography estimation could play an essential role. I've implemented the DeTone method with ResNet34 backbone and got 1.89 test MACE, which is a WAY better than 9.20 reported in the paper with HomographyNet.

What's interesting, in the paper you haven't emphasized enough the best feature of your method in my opinion: outlier robustness. By using RANSAC in the inference phase this method can be robust to outliers / foreground / dynamic objects. And it was confirmed by Le in their Deep Homography Estimation for Dynamic Scenes paper.

Thank you one more time for the discussion.

P.S. I won't close the issue, since the problem is not solved so it can help anyone interested in using your repo.

Best,
Daniel

Hi Daniel,

Thank you so much for your comments. The contributions of this paper come from two aspects (1) perspective field and (2) RANSAC, which provide a new perspective for deep-learning-based homography estimation methods. These features make the proposed network robust to outliers which are commonly seen in real world scenarios. I admit that this paper still has large room for improvement. Thanks for your constructive feedback. Please feel free to leave any discussion in this repo. If you could share your reimplementation of this paper in your own repo, I would appreciate.

Best regards,
Rui Zeng

[dkoguciuk]

Hi Rui,

I am not ready yet, but for sure I'll be trying to advertise the work I'm currently doing ;) Will let you know about the progress 😄

Best,
Daniel

[dkoguciuk]

Hi @ruizengalways ,

as promised HERE is my paper about unsupervised homography estimation (accepted to CVPR2021 workshop), where your PFNet is one of the state-of-the-art methods. Using your perspective field backbone trained in an unsupervised way with perceptual loss gives even better accuracy than training it with supervised loss!

Best,
Daniel

[ruizengalways]

Hi @ruizengalways ,

as promised HERE is my paper about unsupervised homography estimation (accepted to CVPR2021 workshop), where your PFNet is one of the state-of-the-art methods. Using your perspective field backbone trained in an unsupervised way with perceptual loss gives even better accuracy than training it with supervised loss!

Best,
Daniel

Congrats! I carefully read through your paper and fond that this is a really interesting paper. I'd like to ask for a favor. Would it be possible for you share your code and PFNet code in your repository. I would like to link this repository to your PFNet implementation and your work. Thanks !

Best,
Rui

[dkoguciuk]

Hi @ruizengalways ,

yes, I am working on it, but cannot promise anything (the code-sharing policy of my company is a bit tricky). I will get back to you if I succeed.

Best!
Daniel