多址接入技术是移动通信系统实现多用户通信的关键。传统的正交多址接入（Orthogonal Multiple Access，OMA）技术可支持的用户数受限于正交资源数，难以满足未来移动通信系统对海量连接和高频谱效率的需求。为此，业界提出了非正交多址接入（Non-Orthogonal Multiple Access，NOMA）技术。NOMA通过引入可控的干扰，以非正交资源复用的方式有效提高连接密度和频谱效率。然而，NOMA从理论走向实用仍面临着诸多技术挑战，比如上行调度传输时延和信令开销过高、NOMA在多入多出（Multiple-Input Multiple-Output，MIMO）系统中的多维资源难以进行联合优化设计等。为此，本文从未来移动通信系统的需求出发，针对大连接高谱效的NOMA系统优化设计展开深入研究。 首先，针对海量连接场景中上行调度传输时延和信令开销过大的难题，基于压缩感知理论设计低时延低开销的上行免调度NOMA传输机制。一方面，对于基于帧结构的传输系统，挖掘活跃用户的结构化稀疏性，提出基于结构化压缩感知的活跃用户与数据联合检测算法；另一方面，对于突发传输系统，挖掘活跃用户的时间相关性，提出基于动态压缩感知的活跃用户与数据联合检测算法。所提方案可有效提高活跃用户检测精度，有助于实现上行免调度NOMA传输。 然后，针对海量连接场景中低功耗用户设备能量受限的问题，提出基于无线携能的MIMO-NOMA传输机制，基站端利用多天线提供空间复用增益来进一步增大连接数，用户端利用功率分割接收机从接收到的射频信号中同时实现信号检测与能量收割，从而延长低功耗设备的使用寿命。进一步，通过功率分配与功率分割因子联合优化设计实现系统频谱效率和能量效率的折中。所提方案可获得比基于无线携能的MIMO-OMA更高的频谱效率和能量效率。 最后，针对热点高容量场景中毫米波大规模MIMO用户数不能超过射频数的基本限制，提出基于透镜天线的波束空间MIMO-NOMA传输机制。进一步，在提出的波束空间MIMO-NOMA机制中，首先基于波束等效信道设计预编码，然后通过最大化系统可达和速率对所有用户的功率分配进行联合优化设计，并提出低复杂度的迭代优化算法求解非凸功率优化问题。所提方案突破了用户数受限于射频数的限制，可获得比波束空间MIMO更高的频谱效率和能量效率。 本文研究成果为解决NOMA从理论走向实用面临的挑战提供了重要参考。

Multiple access technology is essential for multi-user multiplexing in wireless communication systems. The number of users that can be supported by traditional orthogonal multiple access (OMA) technology is limited by the number of orthogonal resources, which is difficult to meet the requirements of future wireless communications for massive connectivity and high spectrum efficiency. To this end, non-orthogonal multiple access (NOMA) has been proposed. By introducing controllable interference, NOMA can effectively improve connection density and spectrum efficiency through non-orthogonal resource multiplexing. However, NOMA still faces many technical challenges from theory to practicality. For example, delay and signaling overhead in the traditional uplink scheduling transmission are too high, and it is difficult to carry out joint optimization design for NOMA in multiple-input multiple-output (MIMO) systems. For this reason, this thesis starts from the requirements of future wireless communications, and conducts in-depth research on the design of NOMA systems. Firstly, to solve the problem of excessive delay and signaling overhead of traditional uplink scheduling in massive connectivity scenario, the uplink grant-free NOMA scheme having low latency and low overhead is designed based on the theory of compressive sensing (CS). On the one hand, for the frame-based transmission systems, the structured sparsity of active users is investigated, and a joint active user and data detection algorithm based on structured compressive sensing (SCS) is proposed. On the other hand, for the burst transmission systems, the temporal correlation of active users is investigated, and a joint active user and data detection algorithm based on dynamic compressive sensing (DCS) is proposed. The proposed algorithms can effectively improve the detection accuracy of user activity in the uplink grant-free NOMA systems. Then, to solve the problem that the low-power user equipments are energy limited in massive connectivity scenario, a simultaneous wireless information and power transfer (SWIPT)-based MIMO-NOMA scheme is proposed, where SWIPT is implemented by using a power splitting receiver at each user. In this way, signal detection and energy harvesting can be realized simultaneously, which can prolong the service life of the low-power user equipments. Furthermore, a tradeoff between the spectrum efficiency and the energy efficiency is achieved by jointly optimizing power allocation and power splitting factors. The proposed SWIPT-based MIMO-NOMA scheme can achieve higher spectrum efficiency and energy efficiency than those of the SWIPT-based MIMO-OMA scheme. Finally, in hotspot scenario with high data rate, a beamspace MIMO-NOMA scheme based on lens antenna array is proposed, which can break through the fundamental limit of beamspace MIMO systems that the number of users cannot exceed the number of radio frequency (RF) chains. Furthermore, in the proposed beamspace MIMO-NOMA scheme, the precoding is firstly designed based on the equivalent beamspace channel, and then the power allocation of all users is jointly optimized by maximizing the achievable sum rate. To solve the non-convex power optimization problem, a low-complexity iterative optimization algorithm is proposed. The proposed beamspace MIMO-NOMA scheme breaks through the limitation that the number of users is limited by the number of RF chains, and can obtain higher spectrum efficiency and energy efficiency compared with existing beamspace MIMO scheme. The research results in this thesis are expected to provide important reference for solving the technical challenges that NOMA faces from theory to practicality.