手指的灵巧运动和触觉感知是人类完成操作的关键，对机器人而言也是如此。视触觉作为一种利用相机拍摄弹性体形变图像，解析接触信息的新型传感技术，相比传统触觉传感器具有更高的空间分辨率，但在本体设计、信号处理和集成应用中依然面临挑战。本课题基于视触觉传感技术，针对机器人操作的应用场景，设计了TacRot(Tactile and Rotate)灵巧操作末端执行器。TacRot结合了双模态视触觉反馈和主动旋转操作原语，增强了机器人在物体抓取、手内重定向和装配等任务中的灵巧性。 首先，本课题设计了TacRot硬件系统：针对标记点和接触轮廓两种图像特征互相干扰的问题，提出了光照分时复用的双模态分离方法。通过在弹性体表面采用丝网印刷制备标记点阵列，并旋涂漫反射涂层，同时具备了力感知和接触感知两种模态。针对现有视触觉传感器结构设计复杂，难以与机械手集成的问题，设计了模块化的手指，模仿人类两种手内操作方式提出了基于主动旋转的Pivot和Twist操作原语。课题搭建的实验平台为后续触觉标定、定位和操作实验奠定了基础。 其次，分别针对紫外光(UV)下的标记点图像和可见光(RGB)下的接触轮廓图像，提出了相应的触觉感知算法：由标记点的像素位移场SVD分解平移和旋转分量，标定拟合了接触力、力矩的映射函数；基于光度立体算法重建RGB图像接触区域的三维形貌，通过优化建模函数和多角度光照计算平均深度两种方式提高了重建精度。此外，基于触觉图像和点云完成了手内物体位姿估计：通过接触图像连通域的形态学中心矩求解质心坐标和长轴方向，作为三自由度定位结果。将其作为迭代初值，采用点到面距离的ICP算法配准局部接触点云和已知物体模型点云。最后针对触觉本身的局部性导致的退化问题，基于迭代矩阵特征值的阈值选择可观测的自由度优化位姿估计，对比实验证明了所提方法的有效性。 最后，针对TacRot在夹爪闭合、两侧手指同向和反向旋转的三个操作自由度，设计了一系列抓取和操作实验：为二指平行抓取引入触觉反馈，实现自适应抓取宽度和夹持力控制，在抓取实验中达到94%的成功率；在端水和螺丝刀插孔任务中验证了Pivot原语基于触觉重定向的能力；分析了Twist原语的运动模型和假设条件，并在模仿人手搓陀螺和拧螺钉任务中基于触觉反馈探究了模型中各参数的影响，系统在实际对比中展现了超过人手的操作速度。
Dexterous finger movements and tactile perception are key to the completion of human manipulations, as they are for robots. As a new sensing technology that uses a camera to capture images of elastomer deformation and resolve contact information, visual-tactile sensing has higher spatial resolution than traditional tactile sensors, but still faces challenges in ontology design, signal processing and integration applications. Based on the visual-tactile sensing technology, this thesis designs the TacRot (Tactile and Rotate) dexterous manipulation end-effector for the application scenario of robotic manipulation. TacRot combines bimodal tactile feedback and active rotational manipulation primitives to enhance robot dexterity in tasks such as object grasping, in-hand reorientation and assembly. Firstly, the TacRot hardware system is designed: a dual-modal separation method of light time-division multiplexing is proposed for the problem that two image features, marker points and contact contours, interfere with each other. By producing an array of marker dots by screen-printing on the elastomer surface and spin-coating the diffuse reflective layer, both force perception and contact perception modalities are simultaneously available. To address the problem of complex structural design of existing visual-tactile sensors and the difficulty of integration with robotic hands, a modular finger is designed, and the active rotation-based Pivot and Twist primitives are proposed to mimic two types of human in-hand manipulations. The experimental platform lays the foundation for subsequent tactile calibration, localization and manipulation experiments. Secondly, the corresponding tactile perception algorithms are proposed for the marker point image under ultraviolet (UV) and the contact contour image under visible light (RGB) respectively: the mapping functions of contact force and moment are calibrated and fitted by decomposing the translational and rotational components from the pixel displacement field SVD of the marker dots; the 3D morphology of the contact area of the RGB image is reconstructed based on the photometric stereo algorithm. The reconstruction accuracy is improved by both optimizing the modeling function and calculating the average depth by multi-angle illumination. In addition, the estimation of in-hand object position was completed based on tactile images and point clouds: the center-of-mass coordinates and long-axis direction were solved by morphological central moments of the connected domain of contact images as the result of three-degree-of-freedom localization. This is used as an iterative initial value to align the local contact point cloud and the known object model point cloud using the ICP algorithm of point-to-surface distance. Finally, for the degradation problem caused by the localization of the tactile sense itself, observable degrees of freedom are selected based on the threshold of the iterative matrix eigenvalues to optimize the positional estimation, and the effectiveness of the proposed method is demonstrated by comparison experiments. Finally, a series of grasping and manipulation experiments were designed for TacRot with three operational degrees of freedom in jaw closure, simultaneous and reverse rotation of both fingers: introducing tactile feedback for two-finger parallel grasping to achieve adaptive grasping width and gripping force control, achieving a success rate of 94% in the grasping experiment; The ability of the Pivot primitive based on tactile reorientation was verified in the water carrying and screwdriver inserting tasks; the motion model and assumptions of the Twist primitive were analyzed, and the effects of the parameters in the model were explored based on tactile feedback in the tasks of gyroscopic rolling and screwing imitating human hands. The system showed faster speed than human hands in the actual comparison.
