GraphSAGE (val as input) on collab does not reproduce the leaderboard results

[muhanzhang]

Hi team,

I tried the latest ogb example of collab with validation edges also as input but cannot reproduce the leaderboard results of 0.5463% test Hits@50. The command I use is

python gnn.py --use_valedges_as_input --use_sage

And the result I get are:

Hits@50
All runs:
Highest Train: 88.63 ± 1.42
Highest Valid: 53.22 ± 1.10
Final Train: 88.17 ± 1.75
Final Test: 50.18 ± 1.52

I am using ogb 1.2.3 and pyg 1.6.1. Can you have a look? Thanks.

[rusty1s]

Thanks for letting me know. I also increased the number of epochs in my experiments, but didn't commit that change. I'm sorry! That is fixed now.

[muhanzhang]

Cool, thanks! By the way, another related question is why data.full_adj_t (observed graph with validation edges added) is only used in testing, while during training and validation still data.adj_t (without validation edges added) is used? From the rules, validation edges can be both used for model training and as model input, right?

[weihua916]

Hi! That's correct. The requirement is that all the model hyper-parameters need to be tuned using the validation edges. In principle, if you can completely fix all the hyper-parameters using validation edges (including early stopping epoch), then you can use validation edges for model training as well. In this case, we'd assume your training epoch is the same across 10 runs.

[muhanzhang]

Thanks! I have one question about the early stopping epoch. Why do we also need to fix that in order to use validation edges in training? Let's say I use validation edges in training, but still use these validation edges' performance for selecting the best epoch. Although it is less meaningful to do so (because the validation performance is likely to be close to 100% in this setting), I still find it helpful in determining the best epoch than directly using the last epoch. Is this practice allowed?

I am proposing this question because I believe most previous submissions of collab (before allowing the use of validation edges in training) didn't tune epochs explicitly. If we allow the above practice, all previous submissions can directly inherit their tuned hyperparameters in the new setting of using validation in training, and get the new results quickly and consistently. Otherwise, the old and new settings will require two different sets of hyperparameters to tune (one without epochs and one with epochs), thus incurring inconsistencies and extra effort.

[weihua916]

Hi! These are great points. To your first point, that might be based on the observation that, for the ogbl-collab dataset, training longer is the key to achieve good validation accuracy. That's why early stopping is almost unnecessary and training accuracy is a good proxy for validation accuracy. That's a bit of a hack, but again, as long as you follow the rules, that's fine to do. Please do report the "correct" validation accuracy that you've obtained without using training labels (which means you will need to train two separate models, one for reporting validation accuracy, another for reporting test accuracy).

For ogbl-collab, there are not many external submissions. We will keep the leaderboard as it is, and update the leaderboard upon requests from existing submissions. I'd suggest you to make a Google form submission again based on the new rules, and I can delete your previous submission (done under the previous rules) if you want.

Sorry for changing the rules; we hope this would benefit the community in the long run (by allowing time-series modeling). Thanks, and looking forward to your new submission!

[muhanzhang]

Cool! I will make a new submission of SEAL. This submission uses exactly the same hyperparameters tuned with the old rule, except for using validation edges in training. And as a result, the reported validation performance will be the same as the old one.

As for deleting the old result, I notice that currently GCN and GraphSAGE both have two results shown in parallel (one saying "val as input"). I think this is good in the transition phase for giving people a notification that the rule has changed. So I'd like to keep both the old and new results of SEAL too, for now. But please feel free to delete the old results in the future when the community has entirely switched to the new rule :)

[weihua916]

Sounds good!

As for deleting the old result, I notice that currently GCN and GraphSAGE both have two results shown in parallel (one saying "val as input"). I think this is good in the transition phase for giving people a notification that the rule has changed. So I'd like to keep both the old and new results of SEAL too, for now. But please feel free to delete the old results in the future when the community has entirely switched to the new rule :)

This is a great idea and makes a lot of sense. We will delete the old result after a while.

[muhanzhang]

Hi Weihua! I just made a new submission of SEAL (with and without validation as input) to the ogbl-collab leaderboard. It no longer uses any node features and outperforms the AA heuristic, which might suggest that node features are not really needed for this dataset. You may replace the old SEAL results with the new ones. I have also udpated the github with the new hyperparameters. Thanks!

[weihua916]

Hi Muhan, thanks for your submissions! That's an interesting observation. I have updated the leaderboards: I deleted the old SEAL submissions on ogbl-collab. I added the new submissions of SEAL, where I renamed it to SEAL-nofeat just to emphasize that you do not use any node features.

[muhanzhang]

Cool that makes sense. Thank you!