Fetching the paper…

Reading the bibliography…

2020

PointAR: Efficient Lighting Estimation for Mobile Augmented Reality

Zhao, Yiqin, Guo, Tian

Original
PDF
Post on X

Understand

We propose an efficient lighting estimation pipeline that is suitable to run on modern mobile devices, with comparable resource complexities to state-of-the-art mobile deep learning models.

Our pipeline, PointAR, takes a single RGB-D image captured from the mobile camera and a 2D location in that image, and estimates 2nd order spherical harmonics coefficients.
This estimated spherical harmonics coefficients can be directly utilized by rendering engines for supporting spatially variant indoor lighting, in the context of augmented reality.
Our key insight is to formulate the lighting estimation as a point cloud-based learning problem directly from point clouds, which is in part inspired by the Monte Carlo integration leveraged by real-time spherical harmonics lighting.

PointAR: Efficient Lighting Estimation for Mobile Augmented Reality · Around

Built on

Ramamoorthi, R., Hanrahan, P.: An efficient representation for irradiance environment maps. In: Proceedings of the 28th annual conference on Computer graphics and interactive techniques - SIGGRAPH ’01. pp. 497–500. ACM Press, Not Known (2001). https://doi.org/10.1145/383259.383317, http://portal.acm.org/citation.cfm?doid=383259.383317
2001
Earlier work this paper cites.
Chuang, Y.Y.: Camera calibration (2005)
2005
Earlier work this paper cites.
Debevec, P.: Image-based lighting. In: ACM SIGGRAPH 2006 Courses, pp. 4–es (2006)
2006
Earlier work this paper cites.
Gruber, L., Richter-Trummer, T., Schmalstieg, D.: Real-time photometric registration from arbitrary geometry. In: 2012 IEEE international symposium on mixed and augmented reality (ISMAR). pp. 119–128. IEEE (2012)
2012
Earlier work this paper cites.
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385 (2015)
Original
2015
Earlier work this paper cites.
Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., Xiao, J.: 3d shapenets: A deep representation for volumetric shapes. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1912–1920 (2015)
2015
Earlier work this paper cites.
Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., Keutzer, K.: Squeezenet: Alexnet-level accuracy with 50x fewer parameters and ¡0.5mb model size (2016), http://arxiv.org/abs/1602.07360 , cite arxiv:1602.07360Comment: In ICLR Format
Original
2016
Earlier work this paper cites.
Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation. arXiv preprint arXiv:1612.00593 (2016)
Original
2016
Earlier work this paper cites.
Armeni, I., Sax, A., Zamir, A.R., Savarese, S.: Joint 2D-3D-Semantic Data for Indoor Scene Understanding. ArXiv e-prints (Feb 2017)
2017
Earlier work this paper cites.
Chang, A., Dai, A., Funkhouser, T., Halber, M., Niessner, M., Savva, M., Song, S., Zeng, A., Zhang, Y.: Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158 (2017)
Original
2017
Earlier work this paper cites.

Similar

Gardner, M., Sunkavalli, K., Yumer, E., Shen, X., Gambaretto, E., Gagné, C., Lalonde, J.: Learning to predict indoor illumination from a single image. ACM Transactions on Graphics (2017)
2017
Cited alongside, same era.
Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications (2017), http://arxiv.org/abs/1704.04861 , cite arxiv:1704.04861
Original
2017
Cited alongside, same era.
Sze, V., Chen, Y., Yang, T., Emer, J.S.: Efficient processing of deep neural networks: A tutorial and survey. CoRR (2017), http://arxiv.org/abs/1703.09039
Original
2017
Cited alongside, same era.
Cheng, D., Shi, J., Chen, Y., Deng, X., Zhang, X.: Learning scene illumination by pairwise photos from rear and front mobile cameras. Comput. Graph. Forum 37
2018
Cited alongside, same era.

Then

Garon, M., Sunkavalli, K., Hadap, S., Carr, N., Lalonde, J.: Fast spatially-varying indoor lighting estimation. CVPR (2019)
2019
Later among the works it cites.
Liu, W., Sun, J., Li, W., Hu, T., Wang, P.: Deep learning on point clouds and its application: A survey. Sensors 19
2019
Later among the works it cites.
Prakash, S., Bahremand, A., Nguyen, L.D., LiKamWa, R.: GLEAM: An illumination estimation framework for real-time photorealistic augmented reality on mobile devices. In: Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services. pp. 142–154. MobiSys ’19, Association for Computing Machinery, New York, NY, USA (Jun 2019)
2019
Later among the works it cites.
Song, S., Funkhouser, T.: Neural illumination: Lighting prediction for indoor environments. CVPR (2019)
2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Li, Y., Bu, R., Sun, M., Wu, W., Di, X., Chen, B.: Pointcnn: Convolution on x-transformed points. In: Advances in neural information processing systems. pp. 820–830 (2018)

2018

Cited alongside, same era.

Xu, Y., Fan, T., Xu, M., Zeng, L., Qiao, Y.: Spidercnn: Deep learning on point sets with parameterized convolutional filters. arXiv preprint arXiv:1803.11527 (2018)

2018

Cited alongside, same era.

Zhang, E., Cohen, M.F., Curless, B.: Discovering point lights with intensity distance fields. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6635–6643 (2018)

2018

Cited alongside, same era.

Apicharttrisorn, K., Ran, X., Chen, J., Krishnamurthy, S.V., Roy-Chowdhury, A.K.: Frugal following: Power thrifty object detection and tracking for mobile augmented reality. In: Proceedings of the 17th Conference on Embedded Networked Sensor Systems. pp. 96–109. SenSys ’19, ACM, New York, NY, USA (2019)

2019

Cited alongside, same era.

Gardner, M.A., Hold-Geoffroy, Y., Sunkavalli, K., Gagne, C., Lalonde, J.F.: Deep parametric indoor lighting estimation. In: The IEEE International Conference on Computer Vision (ICCV) (October 2019)

2019

Cited alongside, same era.

TensorFlow Mobile and IoT Hosted Models. https://www.tensorflow.org/lite/guide/hosted_models

Cited in the paper.

Google: https://developers.google.com/ar/develop/unity/light-estimation/developer-guide-unity

Cited in the paper.

Liu, L., Li, H., Gruteser, M.: Edge assisted real-time object detection for mobile augmented reality. In: The 25th Annual International Conference on Mobile Computing and Networking (MobiCom’19)

Cited in the paper.

Wu, W., Qi, Z., Fuxin, L.: Pointconv: Deep convolutional networks on 3d point clouds. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)

2019

Later among the works it cites.

ARCore. https://developers.google.com/ar , accessed: 2020-3-3

2020

Closest in time.

Apple: Adding realistic reflections to an AR experience. https://developer.apple.com/documentation/arkit/adding_realistic_reflections_to_an_ar_experience , accessed: 2020-7-10

2020

Closest in time.

Apple Inc: Augmented reality - apple developer. https://developer.apple.com/augmented-reality/ , accessed: 2020-3-3

2020

Closest in time.

Tulloch, A., Gkioxari, G., He, K., Reblitz-Richardson, O., Zhang, P., Vajda, P., Dollar, P., Chatterjee, P., Girshick, R., Tsai, S., Cheung, V., Wang, Y., Jia, Y., He, Z.: Enabling full body AR with mask R-CNN2Go - facebook research. https://research.fb.com/blog/2018/01/enabling-full-body-ar-with-mask-r-cnn2go/ (Jan 2018), accessed: 2020-3-3

2020

Closest in time.