CS348I: Computer Graphics in the Era of AI

Fall 2020, Mon Wed 2:30--3:50pm, Zoom (via Canvas)

teaser

Description

This course introduces deep learning methods and AI technologies applied to four main areas of Computer Graphics: rendering, geometry, animation, and computational photography. We will study a wide range of problems on content creation for images, shapes, and animations, recently advanced by deep learning techniques. For each problem, we will understand its conventional solutions, study the state-of- the-art learning-based approaches, and critically evaluate their results as well as the impacts to researchers and practitioners in Computer Graphics. The topics include differentiable rendering/neural rendering, BRDF estimation, texture synthesis, denoising, procedural modeling, mesh segmentation, view prediction, colorization, style transfer, sketch simplification, character animation, physics simulation, and facial animation. Through programming projects and homework, students who successfully complete this course will be able to use neural rendering algorithms for image manipulation, to apply neural procedural modeling for shape and scene synthesis, to implement policy learning algorithms for creating character animation, and to exploit data-driven methods for simulating physical phenomena.

Prerequisites: CS229, CS231N, or an equivalent intro machine learning course. CS148/248 is recommended but not required.

Announcement: Fall 2020 students, please join Piazza for discussion. All future announcements will be made through Piazza. We will also have polls on Piazza to help people know each other and find teammates.

Staff

C. Karen Liu
Instructor
Jiajun Wu
Instructor
Xinru (Lucy) Hua
CA

Office Hours: The schedule and Zoom links are available on Canvas.
  • Karen: Monday 1:30PM--2:30PM PT during the weeks that Karen lectures.
  • Jiajun: Friday 11AM--noon PT during the weeks that Jiajun lectures.
  • Xinru (Lucy): Thursday 9:00AM--noon PT every week. If you want to request office hours other than this time, please contact Lucy at huaxinru at stanford.edu.

Schedule

Date Lecture Instructor Pset
09/14/2020 Introduction
80 mins lecture
Karen Liu
09/16/2020 Imaging: classic techniques
80 mins lecture
Jiajun Wu
09/21/2020 Imaging (low-level)
40 mins lecture + 40 mins paper presentation and discussion
Jiajun Wu
Required readings (Choose 1 to write a review from):
Image Processing: Exposure: A White-Box Photo Post-Processing Framework
Yuanming Hu, Hao He, Chenxi Xu, Baoyuan Wang, Stephen Lin
TOG 2018
Relighting & View Synthesis: Neural Light Transport for Relighting and View Synthesis
Xiuming Zhang, Sean Fanello, Yun-Ta Tsai, Tiancheng Sun, Tianfan Xue, Rohit Pandey, Sergio Orts-Escolano, Philip Davidson, Christoph Rhemann, Paul Debevec, Jonathan T. Barron, Ravi Ramamoorthi, William T. Freeman
arXiv 2020
Texture Synthesis: Non-Stationary Texture Synthesis by Adversarial Expansion
Yang Zhou, Zhen Zhu, Xiang Bai, Dani Lischinski, Daniel Cohen-Or, Hui Huang
SIGGRAPH 2018
Frame Interpolation: Context-aware Synthesis for Video Frame Interpolation
Simon Niklaus, Feng Liu
CVPR 2018
Deep Nets as Priors: Deep Image Prior
Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky
CVPR 2018

Optional:
ToFlow: Video Enhancement with Task-Oriented Flow
Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, William T. Freeman
IJCV 2019
Colorization: Colorful Image Colorization
Richard Zhang, Phillip Isola, Alexei A. Efros
ECCV 2016
09/23/2020 Imaging (high-level)
40 mins lecture + 40 mins paper presentation and discussion
Jiajun Wu
Required readings (Choose 1 to write a review from):
Image Translation: Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros
ICCV 2017
Image Manipulation: Semantic Photo Manipulation with a Generative Image Prior
David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, Antonio Torralba
SIGGRAPH 2019
Video Inpainting: Flow-edge Guided Video Completion
Chen Gao, Ayush Saraf, Jia-Bin Huang, Johannes Kopf
ECCV 2020
Interactive Video Generation: Vid2Player: Controllable Video Sprites that Behave and Appear like Professional Tennis Players
Haotian Zhang, Cristobal Sciutto, Maneesh Agrawala, Kayvon Fatahalian
arXiv 2020
Style Transfer: A Style-Based Generator Architecture for Generative Adversarial Networks
Tero Karras, Samuli Laine, Timo Aila
CVPR 2019

Optional:
Gated Convolution: Free-Form Image Inpainting with Gated Convolution
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, Thomas Huang
ICCV 2019
EdgeConnect: EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Z. Qureshi, Mehran Ebrahimi
arXiv 2019
Style Transfer: Image Style Transfer Using Convolutional Neural Networks
Leon A. Gatys, Alexander S. Ecker, Matthias Bethge
CVPR 2016
09/28/2020 Rendering: classic rendering techniques
80 mins lecture
Karen Liu HW1 Out: Learning to Render
09/30/2020 Rendering: differentiable rendering
40 mins lecture + 40 mins paper presentation and discussion
Jiajun Wu
Required survey: Differentiable Rendering: A Survey
Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka, Wadim Kehl and Adrien Gaidon
arXiv 2020

Required readings (Choose 1 to write a review from):
Soft Rasterizer: Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning
Shichen Liu, Tianye Li, Weikai Chen, Hao Li
ICCV 2019
DIB-R: Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
Wenzheng Chen, Jun Gao, Huan Ling, Edward J. Smith, Jaakko Lehtinen, Alec Jacobson, Sanja Fidler
NeurIPS 2019
Diffierentiable Ray Tracing: Differentiable Monte Carlo Ray Tracing through Edge Sampling
Tzu-Mao Li, Miika Aittala, Frédo Durand, Jaakko Lehtinen
SIGGRAPH Asia 2018
Differentiable Sphere Tracing: DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing
Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, Zhaopeng Cui
CVPR 2020
Implicit Differentiable Renderer: Multiview Neural Surface Reconstruction with Implicit Lighting and Material
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronen Basri, Yaron Lipman
arXiv 2020

Optional:
OpenDR: OpenDR: An Approximate Differentiable Renderer
Matthew M. Loper, Michael J. Black
ECCV 2014
Neural Mesh Renderer: Neural 3D Mesh Renderer
ddHiroharu Kato, Yoshitaka Ushiku, Tatsuya Harada
CVPR 2018
Differentiable Volumetric Rendering: Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision
Michael Niemeyer, Lars Mescheder, Michael Oechsle, Andreas Geiger
CVPR 2020
10/05/2020 Rendering: neural rendering
40 mins lecture + 40 mins paper presentation and discussion
Jiajun Wu
Required survey: State of the Art on Neural Rendering
Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B Goldman, Michael Zollhöfer
Eurographics STAR 2020

Required readings (Choose 1 to write a review from):
NeRF in the Wild: NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections
Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, Daniel Duckworth
arXiv 2020
Scene Representation Network: Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations
Vincent Sitzmann, Michael Zollhöfer, Gordon Wetzstein
NeurIPS 2019
Deferred Neural Rendering: Deferred Neural Rendering: Image Synthesis using Neural Textures
Justus Thies, Michael Zollhöfer, Matthias Nießner
SIGGRAPH 2019
Neural Reflectance Fields: Neural Reflectance Fields for Appearance Acquisition
Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, Ravi Ramamoorthi
arXiv 2020

Optional:
Visual Object Networks: Visual Object Networks: Image Generation with Disentangled 3D Representation/
Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Joshua B. Tenenbaum, William T. Freeman
NeurIPS 2018
HoloGAN: HoloGAN: Unsupervised learning of 3D representations from natural images
Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, Yong-Liang Yang
ICCV 2019
Neural Volumes: Neural Volumes: Learning Dynamic Renderable Volumes from Images
Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, Yaser Sheikh
SIGGRAPH 2019
Neural Voxel Renderer: Neural Voxel Renderer: Learning an Accurate and Controllable Rendering Tool
Konstantinos Rematas, Vittorio Ferrari
CVPR 2020
Neural Radiance Fields: NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng
ECCV 2020
10/07/2020 Rendering (guest lecture): Implicit Neural Scene Representations
80 mins lecture
Vincent Sitzmann
10/12/2020 Imaging (guest lecture): Understanding and Rewriting GANs
80 mins lecture
Jun-Yan Zhu HW1 Due
10/14/2020 Project proposal discussion Karen Liu, Jiajun Wu Project proposal due 10/13
10/19/2020 Geometry: 3D shape representations, 3D reconstruction Jiajun Wu
10/21/2020 Geometry: neural 3D shape modeling and reconstruction
40 mins lecture + 40 mins paper presentation and discussion
Jiajun Wu
Required readings (Choose 1 to write a review from):
PointNet++: PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas
NeurIPS 2017
Octree Generating Networks: Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs
Maxim Tatarchenko, Alexey Dosovitskiy, Thomas Brox
ICCV 2017
Local Implicit Functions: Local Deep Implicit Functions for 3D Shape
Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, Thomas Funkhouser
CVPR 2020
Deep Signed Distance Function: DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, Steven Lovegrove
CVPR 2019
AtlasNet: AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation
Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, Mathieu Aubry
CVPR 2018

Optional:
3D-GAN: Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
Jiajun Wu, Chengkai Zhang, Tianfan Xue, William T. Freeman, and Joshua B. Tenenbaum
NeurIPS 2016
Point Cloud GAN: Learning Representations and Generative Models for 3D Point Clouds
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, Leonidas Guibas
ICML 2018
Point Set Generation Networks: A Point Set Generation Network for 3D Object Reconstruction from a Single Image
Haoqiang Fan, Hao Su, Leonidas Guibas
CVPR 2017
Generalizable Reconstruction: Learning to Reconstruct Shapes from Unseen Classes
Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Joshua B. Tenenbaum, William T. Freeman, Jiajun Wu
NeurIPS 2018
Occupancy Networks: Occupancy Networks: Learning 3D Reconstruction in Function Space
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, Andreas Geiger
CVPR 2019
10/26/2020 Geometry: neural generative models of 3D structures
40 mins lecture + 40 mins paper presentation and discussion
Jiajun Wu
Required Survey: Learning Generative Models of 3D Structures
Siddhartha Chaudhuri, Daniel Ritchie, Jiajun Wu, Kai Xu, Hao (Richard) Zhang
Eurographics STAR 2020

Required readings (Choose 1 to write a review from):
Beta Shape Machine: Analysis and synthesis of 3D shape families via deep-learned generative models of surfaces
Haibin Huang, Evangelos Kalogerakis, Benjamin Marlin
Eurographics 2015
Shape Structure Generation (StructureNet): StructureNet: Hierarchical Graph Networks for 3D Shape Generation
Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, Leonidas J. Guibas
SIGGRAPH Asia 2019
ShapeAssembly: ShapeAssembly: Learning to Generate Programs for 3D Shape Structure Synthesis
R. Kenny Jones, Theresa Barton, Xianghao Xu, Kai Wang, Ellen Jiang, Paul Guerrero, Niloy J. Mitra, Daniel Ritchie
SIGGRAPH Asia 2020
Configurable Scene Synthesis: Configurable 3D Scene Synthesis and 2D Image Rendering with Per-Pixel Ground Truth using Stochastic Grammars
Chenfanfu Jiang, Siyuan Qi, Yixin Zhu, Siyuan Huang, Jenny Lin, Lap-Fai Yu, Demetri Terzopoulos, Song-Chun Zhu
IJCV 2018
3D Layout Generation: PlanIT: Planning and Instantiating Indoor Scenes with Relation Graph and Spatial Prior Networks
Kai Wang, Yuan Lin, Ben Weissmann, Manolis Savva, Angel Chang, Daniel Ritchie
SIGGRAPH 2019

Optional:
Shape Structure Generation (GRASS): GRASS: Generative Recursive Autoencoders for Shape Structures
Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, Leonidas Guibas
SIGGRAPH 2017
Shape Structure Generation (SCORES): SCORES: Shape Composition with Recursive Substructure Priors
Chenyang Zhu, Kai Xu, Siddhartha Chaudhuri, Renjiao Yi, Hao Zhang
SIGGRAPH Asia 2018
Scene Generation (GRAINS): GRAINS: Generative Recursive Autoencoders for INdoor Scenes
Manyi Li, Akshay Gadi Patil, Kai Xu, Siddhartha Chaudhuri, Owais Khan, Ariel Shamir, Changhe Tu, Baoquan Chen, Daniel Cohen-Or, Hao Zhang
TOG 2019
Shape Program: Learning to Infer and Execute 3D Shape Programs
Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T. Freeman, Joshua B. Tenenbaum, Jiajun Wu
ICLR 2019
10/28/2020 Geometry (guest lecture): Deep 3D Geometry Processing
80 mins lecture
Eric Yi
11/02/2020 Animation: classic animation techniques
80 mins lecture
Karen Liu HW2 Out: Learning to walk
11/04/2020 Animation: data-driven motion synthesis
40 mins lecture + 40 mins paper presentation and discussion
Karen Liu
Required readings (Choose 1 to write a review from):
PFNN: Phase-Functioned Neural Networks for Character Control
Daniel Holden, Taku Komura, Jun Saito
SIGGRAPH 2017
Local Motion Phases: Local Motion Phases for Learning Multi-Contact Character Movements
Sebastian Starke, Yiwei Zhao, Taku Komura, Kazi Zaman
SIGGRAPH 2020
Motion VAEs: Character Controllers using Motion VAEs
Hung Yu Ling, Fabio Zinno, George Cheng, Michiel van de Panne
SIGGRAPH 2020
11/09/2020 Animation: learning motor skills
40 mins lecture + 40 mins paper presentation and discussion
Karen Liu
Required readings (Choose 1 to write a review from):
DeepMimic: DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
Xue Bin Peng, Pieter Abbeel, Sergey Levine, Michiel van de Panne
SIGGRAPH 2018
Learning Locomotion: Learning Symmetric and Low-energy Locomotion
Wenhao Yu, Greg Turk, C.Karen Liu
SIGGRAPH 2018
Learning to control muscles: Scalable Muscle-actuated Human Simulation and Control
Seunghwan Lee, Kyoungmin Lee, Moonseok Park, Jehee Lee1
SIGGRAPH 2019
11/11/2020 Physics Simulation: data-driven physics and differentiable physics
40 mins lecture + 40 mins paper presentation and discussion
Karen Liu
Required readings (Choose 1 to write a review from):
DeepFluid: Deep Fluids: A Generative Network for Parameterized Fluid Simulations
Byungsoo Kim, Vinicius C. Azevedo, Nils Thuerey, Theodore Kim, Markus Gross, Barbara Solenthaler
Eurographics 2019
OptNet: OptNet: Differentiable Optimization as a Layer in Neural Networks
Brandon Amos, J. Zico Kolter
ICML 2017
DiffTaichi: DiffTaichi: Differentiable Programming for Physical Simulation
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, Frédo Durand
ICLR 2020
11/16/2020 Physics Simulation: sim-to-real
80 mins lecture
Jie Tan
Learning Agile Robotic Locomotion: Learning Agile Robotic Locomotion Skills by Imitating Animals
Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang-Wei Edward Lee, Jie Tan, Sergey Levine
RSS 2020
11/18/2020 Final project presentation Karen Liu, Jiajun Wu Project report due 11/17
11/20/2020 HW2 Due