Concerns about the HOIQA dataset

[Gidohub]

Dear Authors,

Thank you very much for providing access to the HOIQA dataset and the accompanying code. We greatly appreciate your efforts and contribution to this research area. Upon examining the dataset and attempting to run evaluations, we have encountered a few discrepancies and would like to request your clarification on the following points.

---

1. Data Count Discrepancies

We have found that the number of samples mentioned in the paper does not match the quantities derived from the provided JSON files.

From the Paper (referenced figures)

(Referenced image in the paper)

Calculated from the Provided JSON Files

• Epic-Kitchens

  • Total folders (videos): 480
  • Total files (images): 208,584

  Training Data: epic-kitchens-train.json

  • Total QA pairs: 899,257
  • Images: 191,679

  Test Data: epic-test.json

  • Total QA pairs: 9,668
  • Images: 9,662

  Test Data: epic-visor-test.json

  • Total QA pairs: 47,020
  • Images: 7,320

  Image Overlap

  • epic-kitchens-train.json & epic-test.json: 0
  • epic-kitchens-train.json & epic-visor-test.json: 72
  • epic-test.json & epic-visor-test.json: 5

  There is a small amount of overlap between epic-visor-test.json and the other datasets. However, it remains unclear which Epic-Kitchens dataset should be used for evaluation, given the discrepancies and lack of explicit instructions.

• Ego4D

  • Total folders (videos): 920
  • Total files (images): 270,005

  Training Data: ego4d-train.json

  • Total QA pairs: 798,663
  • Images: 164,226

  Test Data: ego4d-test.json

  • Total QA pairs: 308,814
  • Images: 105,779

  No image overlap was found between these files.

In both datasets, the number of samples we obtained is lower than the values stated in the paper. Furthermore, for Epic-Kitchens, it is not clear which dataset was intended for evaluation.

---

2. Missing Evaluation Data for Epic-Kitchens

While we were able to run the evaluation code for Ego4D without major issues, there appear to be some problems with the Epic-Kitchens test data, which may require adjustments.

JSON File Structures

• ego4d-test.json provides [bbox] and [noun] fields.

  {"a0705b91-51b7-489d-8b7d-09282f85db6e_9267f964-d5a1-4062-872f-657e2a4cbe81_0_90023-frame_0000090023-lhs": 
  	{"image_path": "Ego4D/v2/frames/9267f964-d5a1-4062-872f-657e2a4cbe81/frame_0000090023.jpg",
      "bbox": 
        [187.02,
        2.62,
        320.48,
        64.1],
      "noun": "left hand"},
   ...
   }

• epic-test.json lacks [bbox] values, causing errors with the default evaluation code.

  {"P01_11_0_57": {
      "frame_path": "EPIC_Kitchens/frames/P01_11/frame_0000000057.jpg",
      "verb": "take",
      "noun": "plate",
      "narration": "take plate"},
    ...
  }

• epic-visor-test.json contains [bbox] values, and we initially considered extracting these values to match the structure of epic-test.json.

  [
    {"question": "[refer] Where is the left hand of the person?",
      "answer": [
        864,
        939,
        1132,
        1080
      ],
      "image_path": "EPIC_Kitchens/frames/P02_09/frame_0000108904.jpg",
      "id": "P02_09_frame_0000108904-001"},
    ...
  ]

However, upon re-examining the overlap, we found only 5 common images between epic-test.json and epic-visor-test.json, suggesting that the vast majority of the data is different. If we choose epic-test.json for evaluation, the required [bbox] coordinates are absent, and since epic-visor-test.json primarily contains different data, it does not serve as a direct substitute. On the other hand, if we opt to use epic-visor-test.json for evaluation, we need to extract data with “[refer]” questions and then derive [bbox] and [noun] fields from the associated question and answer pairs.

---

Request for Clarification

In light of these observations, we kindly request your guidance on the following:

1. Which Epic-Kitchens dataset was originally intended for evaluation, and how should it be handled given the discrepancies in data counts and overlaps?
2. How should we handle the lower-than-expected data counts for both the Epic-Kitchens and Ego4D datasets to ensure proper reproducibility and fairness in evaluation?
3. How can we properly perform the evaluation, especially for Epic-Kitchens, given that epic-test.json lacks [bbox] values and the limited overlap with epic-visor-test.json?

Any advice or additional instructions you could provide would be greatly appreciated. Thank you for your time and consideration.

[Daromog]

Facing the same issue ! Any update on this ?

[Sid2697]

Apologies for the delay on this, I was occupied with other projects. I'll look into this and respond soon. Thanks for understanding!

[Daromog]

Thanks I would appreciate your help on this, currently the dataset has a lot of problems of mismatches pointed by @Gidohub , also answers that dont exist etc, which makes it really difficult to use. Thanks for your efforts on this issue.

[Sid2697]

Hello @Gidohub,

I had a detailed look at your questions and here are my thoughts.

The data counts that you have provided are indeed correct. I checked the json files again and the numbers you reported match. I can't seem to recall now how I got these numbers in the supplementary (as it was done a while ago). But I can assure that if you can use the jsons provided, as these are the ones that we have used. So the comparison will be completely fair. We will update the supplementary to reflect the new numbers. Regarding the overlap in some images from epic-visor-test.json I have noticed that only the images in these cases are common, the questions are different. So the jsons can be used as they are.

Regarding the evaluation process, as noted in the README, one has to first run eval_hoiqa to generate all the predictions and then use evaluate to calculate the numbers. As you can see in:

HOI-Ref/eval_scripts/eval_hoiqa.py

Lines 66 to 109 in 8b9e7cb

	print('Preparing DataLoaders...')
	gt_data = json.load(open(args.gt_json, 'r'))
	ref_data = EPICEvalDataREF(gt_data, vis_processor)
	ref_eval_dataloader = DataLoader(
	ref_data,
	batch_size=args.eval_batch_size,
	shuffle=False,
	num_workers=10,
	)
	ego4d_data = json.load(open(args.ego4d_gt_json, 'r'))
	ego4d_ref_data = EPICEvalDataREF(ego4d_data, vis_processor)
	ego4d_ref_eval_dataloader = DataLoader(
	ego4d_ref_data,
	batch_size=int(args.eval_batch_size/2),
	shuffle=False,
	num_workers=10,
	)
	iden_data = EPICEvalDataIDEN(gt_data, vis_processor)
	iden_eval_dataloader = DataLoader(
	iden_data,
	batch_size=args.eval_batch_size,
	shuffle=False,
	num_workers=10,
	)
	ego4d_iden_data = EPICEvalDataIDEN(ego4d_data, vis_processor, image_original_size=(1440, 1080))
	ego4d_iden_eval_dataloader = DataLoader(
	ego4d_iden_data,
	batch_size=int(args.eval_batch_size/2),
	shuffle=False,
	num_workers=10,
	)
	hoi_gt_data = json.load(open(args.hoi_gt_json, 'r'))
	hoi_data = EPICEvalDataHOI(hoi_gt_data, vis_processor)
	hoi_eval_dataloader = DataLoader(
	hoi_data,
	batch_size=args.eval_batch_size,
	shuffle=False,
	num_workers=10,
	)

we create different data loader for each type of question and generate the output. So you can use the same process for getting the output for each type of question. This handles files with bbox and without bbox entry.
Once you have run eval_hoiqa you can simply run evaluate to calculate the metrics. The evaluate script is written to handle all the different data from epic and ego4d.

Also note that the questions for the test datasets (epic-test.json and ego4d-test.json) are created on the fly by the test data loaders (for example, EPICEvalDataRef and EPICEvalDataIDEN). So you'll need to use these data loaders to get these questions, otherwise, you'll get missing bbox or other missing field issues.

I hope this helps, let me know if there is any confusion. Thanks for your patience on this!

[Daromog]

@Sid2697 Thanks for the previous response, It would be really useful to specify in the jsons which questions are HO-Ref and I-Ref.
I'm personally facing the issue to find the I-Ref samples for B1 and B2, for example I've been searching the tags "refer" and questions like "Locate the object being manipulated by the hand" and others specified in page 25 of the paper, however I'm not able to find any examples of I-Ref in ego4d-train.json and in epic-kitchens-train.json I've found only a few (around 53000 samples).
I would really appreciate your help here.

[Sid2697]

Hello @Daromog,

As mentioned in my previous comment, these questions are generated on the fly using the data loaders as mentioned here. There are two ways you can proceed, a) use these data loaders and save the question answer pairs to use them later or b) integrate these data loaders in your pipeline to generate the questions on the fly.

Also, please refer to the questions distribution in figure 3 of the main paper for the distribution of the questions. I hope this further clarifies your concern.

[cagrigungor]

I noticed an inconsistency in the evaluation datasets. Specifically:
• ego4d-test.json contains bounding box annotations for each sample, and depending on the case, these bounding boxes are used either as part of the question or as part of the answer.
• epic-test.json, however, does not contain any bounding box annotations.

This seems to be an error, because it breaks the evaluation process in both eval_hoiqa.py and evaluate.py. While the percentage of evaluation data affected is relatively small, this issue directly impacts the evaluation code and prevents correct execution.

Could you please update the epic-test.json file with the necessary bounding box annotations to ensure consistency across the datasets?

Thank you very much for your help.