真实感渲染是计算机图形学中的核心研究方向之一，在虚拟现实、影视特效制作、电子游戏和设计可视化等领域发挥着重要作用。随着各行各业对三维内容需求的日益增长，研究面向一般用户的轻量化真实感渲染方法变得愈加重要。传统真实感渲染算法主要面向具有丰富领域知识的专家用户且依赖于繁琐的手工操作，因此存在易用性差和自动化程度低等不足。神经渲染，即基于深度学习的真实感渲染，通过将计算机图形学领域知识和基于大数据的深度学习技术相结合，可以实现更加轻量化的解决方案。 神经渲染研究的重点和难点在于如何合理地将深度学习和真实感渲染领域知识进行结合。本文围绕真实感渲染中采集和建模、存储和表达以及绘制和可视化三个核心领域进行研究，通过充分挖掘问题本身的特性以找到将深度学习有效融入的方式，并提出了一系列全新的神经渲染方法，具体包括：1. 提出了一种基于逆渲染和数据驱动的表观建模方法。该方法以轻量化的采集设备（手机相机）所拍摄的若干图片作为输入，其核心思路是在深度自编码器网络构建的隐空间中进行逆渲染优化。基于深度学习的数据先验信息的融入使得在整个逆渲染优化过程中无需任何手工设计的启发式约束。该方法可以支持任意分辨率和任意数量的输入图片，并且其重建质量随输入图片数量的增加而不断提升，相较于传统方法大大提升了方法适用范围和表观建模效率。2. 提出了一种基于深度场景表达的重光照方法。该方法以两个手持设备拍摄的无结构化图片为输入，可以实现真实世界复杂场景的自由视点重光照渲染。该方法以神经纹理和辐射亮度信息作为深度场景表达并利用神经渲染网络完成重光照渲染。该方法还提出了光照增强策略来扩展神经渲染网络所支持的光源类型。相较于基于建模的传统方法，该方法大大降低了采集工作量和复杂度，并且可以支持包含复杂表观和全局光照效果的复杂真实场景。3. 提出了一种基于深度绘制管线的全局光照绘制方法。该方法提出使用深度全连接网络来建模着色点到全局光照的复杂映射，可支持动态面光源下全频率全局光照的快速渲染。得益于该方法提出的神经网络友好的输入表达，深度全连接网络可以在保证紧凑性的同时有效学习复杂的全局光照效果。该方法还提出了一种基于材质划分的加速策略以进一步提升运行效率和降低存储开销。
Photorealistic rendering plays a crucial role in many computer graphics applications such as movie visual effects, 3D video games, virtual reality, and design visualization. With the increasing demand for 3D content, it is important to develop lightweight photorealistic rendering approaches for general users. Classic photorealistic rendering methods mainly focus on expert users with rich domain knowledge and require expensive manual intervention, and thus have difficulty in applying to lightweight applications. Neural rendering, deep-learning-based photorealistic rendering, is able to achieve lightweight solutions by combining domain knowledge of computer graphics with deep learning techniques. How to combine deep learning with knowledge of photorealistic rendering in an efficient way is the main challenge in neural rendering. This dissertation focuses on the three major directions of photorealistic rendering, i.e. acquisition and modeling, storage and representation, and rendering and visualization. This dissertation proposes several efficient neural rendering approaches by exploiting the characteristics of the specific problems first and then integrating suitable deep learning modules. Specifically, this dissertation proposes:1. A unified framework for estimating high-resolution surface reflectance properties of a spatially-varying planar material sample from an arbitrary number of photographs. This method combines deep learning and inverse rendering in a flexible and easy-to-implement framework that performs the inverse rendering optimization without any handcrafted heuristics in a learned SVBRDF latent space characterized by a fully convolutional auto-encoder. The precision of the estimated appearance scales from plausible approximations when the input images fail to reveal all the reflectance details to accurate reproductions for sufficiently large sets of input images. The proposed unified framework is suitable for estimating high-resolution SVBRDFs from an arbitrary number of input photographs with a lightweight acquisition setup.2. A novel image-based method for 360° free-viewpoint relighting from unstructured photographs of a scene captured with double handheld devices. This method leverages a scene-dependent neural rendering network for relighting a rough geometric proxy with learnable neural textures. The key to making the rendering network lighting-aware are radiance cues: global illumination renderings of a rough proxy geometry of the scene for a small set of basis materials and lit by the target lighting. This method introduces a novel lighting augmentation strategy that exploits the linearity of light transport to extend the relighting capabilities of the neural rendering network to support other lighting types beyond the lighting used during acquisition. Compared to classic model-based approaches, this method can handle more intricate scenes with a wide variety of material properties and global light transport effects and reduce the data acquisition cost. 3. A carefully designed framework for interactively rendering full global illumination with dynamic area light sources. The complex mapping from the input of each shading point to global illumination is modeled by a deep fully-connected network that is well-suited to approximate such complex mapping. The neural-network-friendly input representation plays a crucial role in reducing the requirement of network size without affecting fitting quality. This method supports many realistic global illumination effects such as glossy interreflection, caustics, and color bleeding. This method proposes a material-based partition strategy to further improve the run-time performance and reduce the storage cost.
