System Design and Evaluation of Multi-Modal Magnetic Tactile Sensor for Robotic Grasping

In robotic manipulation and grasping tasks, visual sensing is commonly used to gather information about external object properties, including position, shape, and orientation. However, in unstructured environments where occlusions and low visibility may occur, visual information alone is insufficient for providing the necessary tactile feedback. In contrast, tactile sensors offer critical information regarding an object's pose, texture, and force, thereby enhancing a robotic system's ability to grasp and manipulate objects with greater precision. These sensors enable robots to perform tasks such as delicate grasping, shape recognition, and texture detection. This thesis work, developed at the Institute of Robotics and Mechatronics (RMC) of the German Aerospace Center (DLR), addresses the pose estimation problem of both the fingertips of robotic grippers, as well as the objects being grasped. It emphasizes the role of tactile sensors in object manipulation and grasping operations, particularly in scenarios where visual data alone may be inadequate. Among the tactile sensing technologies developed in recent years—such as capacitive, piezoresistive, and optical sensors—Hall-effect-based tactile sensors are notable for their multi-directional sensing capabilities, relatively low cost, and ease of fabrication. To achieve effective pose estimation for robotic grasping applications, this work explores the firmware design and evaluation of a magnetic-based multi-modal tactile sensor specifically developed by the Institute of Robotics and Mechatronics at DLR. The proposed tactile sensor consists of four Hall-effect sensors and a 9-degree-of-freedom (DoF) inertial measurement unit (IMU). Together, these sensors facilitate responsive measurement of touch-related force vectors while supporting orientation tracking through the fusion of IMU sensor data. It is crucial to overcome magnetic interference and drift effects that can distort readings, which are important considerations in sensor design and calibration. For reliable real-time operation and communication within robotic systems, this work utilizes the Zephyr Real-Time Operating System (RTOS). Since Zephyr's framework lacked built-in sensor drivers for the Hall-effect and IMU sensors, custom drivers were developed in C based on detailed sensor datasheet specifications. This development facilitated sensor management and signal processing within the Zephyr RTOS. The thesis proceeds by analyzing the Hall sensor data independently of the IMU data, conducting separate signal analyses. Calibration and sensor fusion algorithms for the IMU sensors (accelerometer, magnetometer, and gyroscope) allow for the derivation of quaternion or Euler angle information, providing precise orientation of objects in contact with the tactile sensor. Analyses and evaluations of both the tactile and IMU data were performed, yielding a comprehensive overview of the overall functionalities of the tactile sensor. Through this work, a novel tactile sensing approach is proposed that integrates multi-modal force and orientation feedback. This approach enhances the feasibility of pose estimation for robotic grasping, with potential applications in unstructured environments where traditional vision-based systems may not be suitable.