How do I extract the force (along the direction of the gripper) exerted by the end of the gripper on the hemispherical object ? I tried using a force sensor with a site defined at the end of the gripper but it seems to register values even when the end of the gripper doesn't touch anything, also are the xyz axes with respect to the world coordinates or is it with respect to the site ?
You could search through d->contact directly to look for the contact force. If you use mj_contactForce() to find the contact force it will be in contact frame.