触控技术作为手机、平板和智能眼镜等便携设备的核心交互方式，受制于传感器低空间分辨率导致的单位面积输入带宽不足，当前仅能捕捉手指接触点位置信息，无法充分开发人手高自由度的交互潜力，这严重制约了便携设备的交互体验与效率。指纹传感器作为高分辨率接触式传感器已在身份识别领域广泛应用，但其作为新型人机交互传感器的潜力还未被充分挖掘。本文探索基于指纹传感器的手指状态估计方法，研制测量手的姿态、运动轨迹、手势等状态的新技术，提高人机交互的输入带宽，从而为便携计算设备的人机交互提供更丰富自然的信息输入。研究主要围绕以下三方面展开：一、基于直接成像的大范围三维手指姿态估计：手指姿态估计技术有望显著提升触摸屏设备的输入能力，但现有技术依赖的传感器和姿态表示方式存在不足，难以从中推断大范围的三维手指角度。本文提出利用直接光学成像全面采集手指接触与非接触区域的图像，并提出了基于轴角表示的手指姿态估计网络模型。实验结果表明，该方法在全角度范围内实现了准确而鲁棒的姿态估计，将角度估计误差降低了55.8%，为基于手指角度的触摸屏交互方式奠定了基础。二、基于指纹图像序列的大符号集动态手势识别：智能眼镜常用的触控手势识别技术支持的手势少，严重限制了交互效率。本文提出基于嵌入式指纹传感器的交互技术，设计了基于规则和深度学习的手势识别算法，能够从局部指纹图像序列中识别动态运动模式、身份特征及多样手势。实验结果显示该算法在动态手势识别和手指识别任务中分别取得了98.8% 和99.2% 的准确率，以仅相当于物理键盘1/200 的面积实现了键盘功能，有望用于智能眼镜等设备的大符号集输入。三、基于高帧率指纹传感图像的手指运动精细跟踪：触摸板具有优良的指针精确控制和快捷手势识别能力，但过大的面积影响了其在便携计算设备中的应用。本文提出一种快速指纹匹配算法，通过高帧率指纹图像实时捕捉手指细微位移并实现一系列交互功能。实验表明该算法的运动跟踪误差仅0.04 毫米，帧率100+帧/秒，手指识别性能98.5%，仅用2 mm2（常规触摸板的1/1000）实现了触摸板的完整功能，为便携设备的连续指针控制和离散命令输入提供了空间效率高的方案。基于上述技术开发的三个交互系统原型在各项用户实验中表现良好，验证了所提技术在笔记本电脑、手机、智能眼镜等便携计算机设备上的实用性。
Touch technology serves as the core interaction method for portable devices suchas smartphones, tablets, and smart glasses. However, due to the limited spatial resolutionof sensors, the input bandwidth per unit area is insufficient. Currently, these devicescan only capture the positional information of finger touch points, failing to leveragethe high degree of freedom interaction potential of the human hand, which severely restrictsthe interaction experience and efficiency of portable devices. Fingerprint sensors,as high-resolution contact sensors, have been widely used in identity recognition but theirpotential as novel human-computer interaction sensors remains underexplored. This paperexplores methods for estimating finger states based on fingerprint sensors, developingnew technologies to measure hand posture, movement trajectories, gestures, and more, toenhance the input bandwidth of human-computer interaction, thereby providing richer andmore natural information input for portable computing devices. The research primarilyfocuses on three aspects:1. Comprehensive 3D Finger Posture Estimation Based on Direct Imaging. Fingerposture estimation technology promises to significantly enhance the input capabilitiesof touch screen devices. However, existing technologies suffer from limitations in sensorcapabilities and posture representation methods, making it difficult to infer large-scale3D finger angles. This paper proposes a method using direct optical imaging to comprehensivelycapture images of both contact and non-contact areas of the finger, and introducesa network model for finger posture estimation based on axis-angle representation.Experimental results show that this method achieves accurate and robust posture estimationacross the full range of angles, reducing angle estimation error by 55.8%, laying thegroundwork for touch screen interactions based on finger angles.2. Dynamic Gesture Recognition with Large Symbol Sets Based on FingerprintImage Sequences. The gesture recognition technology commonly used in smart glassessupports a limited number of gestures, significantly restricting interaction efficiency. Thispaper proposes an interaction technology based on embedded fingerprint sensors, designinggesture recognition algorithms using rule-based and deep learning approaches thatcan identify dynamic movement patterns, identity features, and various gestures fromsequences of local fingerprint images. Experimental results indicate that the algorithmachieves 98.8% accuracy in dynamic gesture recognition and 99.2% in finger identificationtasks, offering keyboard functionality with just 1/200th of the area of a physicalkeyboard, promising applications for large symbol set input in devices like smart glasses.3. Fine Tracking of Finger Movement Based on High Frame Rate FingerprintSensor Images. Touchpads offer excellent precision control and fast gesture recognition,but their large size affects their application in portable computing devices. This paperpresents a fast fingerprint matching algorithm that captures minute finger movements inreal-time using high frame rate fingerprint images, facilitating a range of interactive functions.Experiments show that the tracking error of this algorithm is only 0.04 mm, with aframe rate above 100 frames per second and 98.5% finger recognition performance, usingjust 2 mm2 (1/1000 of a conventional touchpad) to achieve full touchpad functionality,providing a space-efficient solution for continuous pointer control and discrete commandinput in portable devices.The three interaction system prototypes developed based on the aforementioned technologieshave performed well in various user experiments, validating the practicality ofthese technologies on portable computing devices like laptops, smartphones, and smartglasses.
