轨迹规划作为自动驾驶的核心技术之一，需要高效可靠地生成车辆行驶轨迹。轨迹优化因对轨迹规划任务建模具有准确、直接、统一的优点，近些年得到广泛关注。随着应用场景复杂性增加，轨迹优化三个方面的问题逐渐暴露出来。首先，轨迹优化存在着建模简化和轨迹可跟踪间的矛盾，过于简化的模型将导致无法控制车辆跟踪轨迹。其次，非凸避撞限制和较强非线性的车辆运动特性使优化问题求解十分费时。再次，一些长尾场景存在与场景特性有关的特殊不可微约束，使优化问题求解变得十分困难。为此，论文从有效建模、实时求解、特殊场景约束处理三个方面对复杂环境下自动驾驶轨迹规划优化问题进行研究。针对轨迹规划建模简化性和轨迹可跟踪性的矛盾，论文提出非结构化场景自动驾驶轨迹优化问题建模准则，通过改变优化问题中决策变量及车辆运动方程设定，实现建模简化和轨迹可跟踪性的平衡。同时，为了克服数值优化求解依赖初始解的弊端，论文提出了粗糙轨迹寻找和数值优化提升的两阶段轨迹规划框架，以初始轨迹作引导搜索，使数值求解快速收敛到局部最优。在第一阶段，针对复杂环境初始解搜索复杂度高问题，论文提出了用低维度关键变量表征整条轨迹的初始轨迹寻找方法，通过将初始轨迹的求解问题拆解为关于关键变量的线性求解问题，降低初始轨迹寻找计算复杂度。实验结果表明，该方法能够在较短时间内获得高质量的初始轨迹。在第二阶段，针对非凸避撞约束处理困难问题，论文提出了基于自适应可行空间凸化的处理方法，通过凸走廊处理、自适应系列圆近似和解析扩充方式，将非凸可行集缩为凸可行集。针对非线性运动学约束处理困难问题，论文提出了基于分段轨迹段线性化范数惩罚的运动学约束处理方法，通过线性化范数惩罚处理，将轨迹优化问题转为序列凸优化问题，并采用分段并行操作，减少大规模轨迹规划计算负担。实验结果表明，本文的算法有效提高了轨迹规划求解效率。为解决特殊场景特有约束求解困难问题，论文针对以路肩泊车为代表的不可微区域依赖约束处理困难问题，提出了区域依赖约束近似表征，将不可微约束转为可微约束；针对以感知不确定规划为代表的概率型避撞约束处理困难问题，提出了机会约束确定等价表征，将不确定性避撞约束转为确定性避撞约束。实验结果表明所提方法能够成功解决几类典型复杂的轨迹规划任务。

As one of the core technologies of autonomous driving, trajectory planning requires the efficient and reliable generation of vehicle trajectories. Trajectory optimization has received widespread attention in recent years due to its accurate, direct, and unified modeling of trajectory planning tasks. However, as the complexity of application scenarios increases, problems in three aspects of trajectory optimization are gradually exposed. Firstly, trajectory optimization suffers from the contradiction between modeling simplicity and trajectory trackability, and too simplified models will result in the inability to control the vehicle to track the trajectory. Secondly, non-convex collision avoidance constraints and strong nonlinear vehicle motion characteristics make optimization problem-solving very time-consuming. Thirdly, some long-tail scenarios have special non-differentiable constraints related to scene characteristics, making optimization problem-solving very difficult. To address these issues, this dissertation studies the optimization problems of autonomous driving trajectory planning in complex environments from three aspects: effective modeling, real-time problem solving, and special scene constraint handling.To address the contradiction between modeling simplification and trajectory traceability, this dissertation proposes modeling guidelines for non-structured autonomous driving trajectory optimization problems, balancing modeling simplification and trajectory traceability by changing the decision variables and vehicle motion equation settings in the optimization problem. Additionally, to overcome the drawbacks of numerical optimization solutions that rely on initial guesses, this dissertation proposes a two-stage trajectory planning framework for rough trajectory searching and numerical optimization enhancement, with the initial trajectory as a guide for searching, to enable numerical solution convergence to local optimality quickly.In the first stage, to address the high complexity of initial solution searching in complex environments, this dissertation proposes a method for finding initial trajectories that use low-dimensional key variables to characterize the entire trajectory. By decomposing the initial trajectory search problem into a linear solving problem about key variables, the computational complexity of initial trajectory searching is reduced. Experimental results show that this method can obtain high-quality initial trajectories in a short period.In the second stage, to address the difficult handling of non-convex collision avoidance constraints, this dissertation proposes a processing method based on adaptive feasible space convexification, which reduces non-convex feasible sets to convex feasible sets through convex corridor processing, adaptive circle approximation, and analytical expansion. To address the difficulty of handling non-linear kinematic constraints, this dissertation proposes a kinematic constraint handling method based on segment trajectory linearization and norm penalty, which converts the trajectory optimization problem into a series of convex optimization problems and adopts a segment parallel operation to reduce the computational burden of large-scale trajectory planning. Experimental results show that the proposed algorithms effectively improve trajectory planning solution efficiency.To address the difficult problem of special scene-specific constraints, this dissertation proposes an approximate representation of region-dependent constraints, which converts non-differentiable constraints into differentiable constraints for shoulder parking. Additionally, for handling probabilistic collision avoidance constraints caused by uncertain perception, this dissertation proposes a method for equivalent representation of chance constraints, making uncertain collision avoidance constraints deterministic. Experimental results show that the proposed methods can successfully solve several typical complex trajectory planning tasks.