发展和完善新的实验探测手段、探索和研究新奇的量子现象，不仅有助于推动基础物理学研究，也具有潜在的应用价值。本论文包含两部分工作：一是研制了超高真空扫描微波阻抗显微镜-分子束外延生长联合系统；二是对电荷密度波、本征磁性拓扑绝缘体及氧化物衬底上的纳米尺寸超导体等新奇现象的扫描隧道显微镜研究。 扫描微波阻抗显微镜是一种新兴的扫描探针显微技术，我们将其工作环境拓展到超高真空，并与分子束外延生长、扫描隧道显微镜联合，这将实现样品的原位制备与测量，极大地扩展扫描微波阻抗显微镜的应用范围。目前这台系统已完成搭建和调试，其最低工作温度为450 mK、最高磁场11 T、空间分辨率为50 nm，这些指标均达到国际先进水平。 新型稀土金属碲化物EuTe4不仅展现出电荷密度波，还具有温度跨度超过400 K的非常规热滞转变。我们对EuTe4进行了扫描隧道显微镜研究，首次观测到偏轴的电荷密度调制，其成因可能来自应力与低温下电荷密度波被抑制的共同作用。研究揭示了材料中非常规热滞现象的两种可能机制：一是两种电荷有序态之间的转换，二是缺陷对电荷密度波的钉扎效应。 本征磁性拓扑绝缘体MnBi2Te4的发现为提高量子反常霍尔效应温度提供了新的可能。利用扫描隧道显微镜，我们对其表面电子结构进行了研究。实验中没有观测到拓扑表面态存在的证据，也未观测到表面电子对磁场的响应，我们认为这是由于该材料在解理后发生了结构分层，即表面的磁结构与体材料截然不同。这解释了能带与输运测量结果之间的矛盾，也为改善该体系的磁性提供了重要信息。 FeSe/SrTiO3体系中的界面超导增强现象为高温超导研究提供了新思路。利用分子束外延技术，我们在SrTiO3衬底上制备了纳米尺寸的铅岛，并开展了扫描隧道显微镜研究。实验结果表明，在纳米尺度下，电子的行为具有显著的尺寸相关性。在中等尺寸的铅岛中，观测到超导能隙增大、电子-声子耦合增强、库伦能隙、赝能隙等新奇现象。该体系的超导转变温度与体相铅相比没有提高，可能与尺寸效应导致的库伦相互作用增强有关。
The development of new experimental instruments with extended functionalities, as well as the exploration of novel quantum phenomena are not only critical to the understanding of fundamental physics, but also important for developing new applications. This thesis contains two parts: the first is the construction of an ultra-high vacuum scanning microwave impedance microscope – molecular beam epitaxy combined system, and the second is the study of novel phenomena such as charge density wave, intrinsic magnetic topological insulator, and nanoscale superconductor grown on oxide substrates by scanning tunneling microscope. Scanning microwave impedance microscopy is a relatively new member of scanning probe microscopy family. We have not only successfully extended it to ultra-high vacuum condition, but also combined it with molecular beam epitaxy and scanning tunneling microscopy. The main technical parameters of this system such as minimum operating temperature (450 mK), highest magnetic field (11 T) and spatial resolution (50 nm) have all reached the international leading level. The newly discovered rare-earth telluride compound EuTe4 exhibits not only charge density waves, but also unconventional thermal hysteresis transitions with the temperature span exceeding 400 K. We characterize the CDW state in EuTe4 with scanning tunneling microscopy. We observed off-axis charge density modulations which were absent in previous studies of bulk materials and may originate from the joint effect of strain and suppression of charge density waves at low temperature. Our study reveals two possible mechanisms for the unconventional thermal hysteresis in EuTe4, namely the interaction between two charge orders and the pinning effect of CDW by defects. The discovery of intrinsic magnetic topological insulator MnBi2Te4 have attracted extensive attention. Using scanning tunneling microscopy, we studied the surface electronic structure of MnBi2Te4 single crystal. We found no evidence for the existence of topological surface states, nor did we observe the response of surface states to magnetic fields. We therefore propose that after sample cleavage, the magnetic structure of the surface layer is distinct from that of the bulk material. This model explains not only our experimental observations, but also the contradictory results given by transport and band structure measurements. The discovery of interface enhanced superconductivity in FeSe/SrTiO3 provides a new platform for the study of high-temperature superconductivity. Using molecular beam epitaxy, we obtained nanometer-sized lead islands on SrTiO3 substrates. STM study showed size dependent electron behavior. In medium-sized lead islands, novel phenomena such as enlarged superconducting energy gap, enhanced electron-phonon coupling, Coulomb gap and pseudogap were observed. Unfortunately, the superconducting transition temperature of this system is not boosted compared with bulk lead, which may be related to the enhanced coulomb interaction by size effects.
