wrong about loading model

[zhangjinyangnwpu]

when I want to test with cpu, and I load model as follows, there are some missed key, and the

final result seem not right, can you give some suggesting?

wrong info:

WARNING: You are using a MANO model, with only 10 shape coefficients.
WARNING: You are using a MANO model, with only 10 shape coefficients.
Fix shapedirs bug of MANO
[2023-12-10 19:13:54,189] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to mps (auto detect)
NOTE: Redirects are currently not supported in Windows or MacOs.
Load checkpoint from ./snapshot_6.pth
Traceback (most recent call last):
  File "demo.py", line 58, in <module>
    model.load_state_dict(new_state_dict, strict=True)
  File "/Users/zhangjinyang/opt/anaconda3/envs/torch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for Model:
        Missing key(s) in state_dict: "mano_layer_right.body_pose", "mano_layer_right.shapedirs", "mano_layer_right.faces_tensor", "mano_layer_right.v_template", "mano_layer_right.J_regressor", "mano_layer_right.posedirs", "mano_layer_right.parents", "mano_layer_right.lbs_weights", "mano_layer_right.hand_mean", "mano_layer_right.pose_mean", "mano_layer_right.vertex_joint_selector.extra_joints_idxs", "mano_layer_left.body_pose", "mano_layer_left.shapedirs", "mano_layer_left.faces_tensor", "mano_layer_left.v_template", "mano_layer_left.J_regressor", "mano_layer_left.posedirs", "mano_layer_left.parents", "mano_layer_left.lbs_weights", "mano_layer_left.hand_mean", "mano_layer_left.pose_mean", "mano_layer_left.vertex_joint_selector.extra_joints_idxs". 

I had similar results when using cuda:

WARNING: You are using a MANO model, with only 10 shape coefficients.
WARNING: You are using a MANO model, with only 10 shape coefficients.
Fix shapedirs bug of MANO
>>> Using GPU: 0
Load checkpoint from ./snapshot_6.pth
./snapshot_6.pth
Traceback (most recent call last):
  File "demo.py", line 61, in <module>
    model.load_state_dict(ckpt['network'], strict=True)
  File "/home/xiaobing/miniconda3/envs/ma/lib/python3.8/site-packages/torch/nn/modules/module.py", line 2152, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for DataParallel:
        Missing key(s) in state_dict: "module.mano_layer_right.body_pose", "module.mano_layer_right.shapedirs", "module.mano_layer_right.faces_tensor", "module.mano_layer_right.v_template", "module.mano_layer_right.J_regressor", "module.mano_layer_right.posedirs", "module.mano_layer_right.parents", "module.mano_layer_right.lbs_weights", "module.mano_layer_right.hand_mean", "module.mano_layer_right.pose_mean", "module.mano_layer_right.vertex_joint_selector.extra_joints_idxs", "module.mano_layer_left.body_pose", "module.mano_layer_left.shapedirs", "module.mano_layer_left.faces_tensor", "module.mano_layer_left.v_template", "module.mano_layer_left.J_regressor", "module.mano_layer_left.posedirs", "module.mano_layer_left.parents", "module.mano_layer_left.lbs_weights", "module.mano_layer_left.hand_mean", "module.mano_layer_left.pose_mean", "module.mano_layer_left.vertex_joint_selector.extra_joints_idxs". 

# snapshot load
model_path = './snapshot_6.pth'
assert osp.exists(model_path), 'Cannot find model at ' + model_path
print('Load checkpoint from {}'.format(model_path))
model = get_model('test')
# model = DataParallel(model).cuda()
ckpt = torch.load(model_path, map_location=torch.device('cpu'))
new_state_dict = {k.replace("module.", ""): v for k, v in ckpt['network'].items()}
model.load_state_dict(new_state_dict, strict=True)
# model.load_state_dict(ckpt['network'], strict=True)
model.eval()

when I set strict = False, results like this:

[mks0601]

I think the results are the same as this one, means there is no problem?
https://github.com/facebookresearch/InterWild/tree/main/demo/renders

[zhangjinyangnwpu]

I think the results are the same as this one, means there is no problem? https://github.com/facebookresearch/InterWild/tree/main/demo/renders

yes, but the results I show are not exactly the same as here

[mks0601]

I can't get what you're saying. So when setting strict=False, your results are not the same as here (https://github.com/facebookresearch/InterWild/tree/main/demo/renders)? How do your results look like?

[zhangjinyangnwpu]

I'm sorry for not expressing myself clearly. When loading the model, set strict to fasle, the model can be loaded successfully, and the inference result is like the picture I showed above; When strict is set to false, the model loading fails, and the error message like this:

Traceback (most recent call last):
  File "demo.py", line 61, in <module>
    model.load_state_dict(ckpt['network'], strict=True)
  File "/home/xiaobing/miniconda3/envs/ma/lib/python3.8/site-packages/torch/nn/modules/module.py", line 2152, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for DataParallel:
        Missing key(s) in state_dict: "module.mano_layer_right.body_pose", "module.mano_layer_right.shapedirs", "module.mano_layer_right.faces_tensor", "module.mano_layer_right.v_template", "module.mano_layer_right.J_regressor", "module.mano_layer_right.posedirs", "module.mano_layer_right.parents", "module.mano_layer_right.lbs_weights", "module.mano_layer_right.hand_mean", "module.mano_layer_right.pose_mean", "module.mano_layer_right.vertex_joint_selector.extra_joints_idxs", "module.mano_layer_left.body_pose", "module.mano_layer_left.shapedirs", "module.mano_layer_left.faces_tensor", "module.mano_layer_left.v_template", "module.mano_layer_left.J_regressor", "module.mano_layer_left.posedirs", "module.mano_layer_left.parents", "module.mano_layer_left.lbs_weights", "module.mano_layer_left.hand_mean", "module.mano_layer_left.pose_mean", "module.mano_layer_left.vertex_joint_selector.extra_joints_idxs". 

when strict is set to false, this is the result I infered：

and this is the result provided by this repository：

these two results are not exactly the same

[mks0601]

mano layers are not updated during the training as you can see in here

InterWild/main/model.py

Line 35 in b570c12

self.trainable_modules = [self.body_backbone, self.body_box_net, self.hand_roi_net, self.hand_position_net, self.hand_rotation_net, self.hand_trans_net]

you don't have gpu environment? if you have, could you test with gpu environment?

[linjiangya]

@mks0601 Happy New Year. I didn't have any issue loading the model on my GPU, but also produced the results below, which are the same as @zhangjinyangnwpu 's:

The results below are slightly different from those in your repository /demo/renders/. Moreover, I think the skeleton doesn't seem to align with the MANO mesh.

[mks0601]

skeletons are estimated ones, not from projecting mano meshes. So the skeleton and mano mesh could be different.