When I use Reinforcement learning with CIDEr reward,there is always keyerror mistake

[userpei]

Traceback (most recent call last):
File "train.py", line 171, in
train(opt)
File "train.py", line 102, in train
reward = get_self_critical_reward(decoder, core_args, vrg_data, fc_feats, att_feats, weak_relas, att_masks, data, gen_result, opt)
File "/peng/pyx/WeakVRD-Captioning-master/misc/rewards_graph.py", line 60, in get_self_critical_reward
, cider_scores = CiderD_scorer.compute_score(gts, res)
File "cider/pyciderevalcap/ciderD/ciderD.py", line 48, in compute_score
(score, scores) = cider_scorer.compute_score(self._df)
File "cider/pyciderevalcap/ciderD/ciderD_scorer.py", line 199, in compute_score
score = self.compute_cider(df_mode)
File "cider/pyciderevalcap/ciderD/ciderD_scorer.py", line 173, in compute_cider
vec, norm, length = counts2vec(test)
File "cider/pyciderevalcap/ciderD/ciderD_scorer.py", line 122, in counts2vec
df = np.log(max(1.0, self.document_frequency[ngram]))
KeyError: ('4857', '8713', '344', '993')

The numbers are different each time