Fetching the paper…

Reading the bibliography…

2023

GNOT: A General Neural Operator Transformer for Operator Learning

Hao, Zhongkai, Wang, Zhengyi, Su, Hang et al.

Original
PDF
Post on X

Understand

Learning partial differential equations' (PDEs) solution operators is an essential problem in machine learning.

However, there are several challenges for learning operators in practical applications like the irregular mesh, multiple input functions, and complexity of the PDEs' solution.
To address these challenges, we propose a general neural operator transformer (GNOT), a scalable and effective transformer-based framework for learning operators.
By designing a novel heterogeneous normalized attention layer, our model is highly flexible to handle multiple input functions and irregular meshes.

GNOT: A General Neural Operator Transformer for Operator Learning · Around

Built on

Introduction to partial differential equations with applications
Zachmanoglou, E. C. and Thoe, D. W · 1986
Earlier work this paper cites.
Adaptive mixtures of local experts
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E · 1991
Earlier work this paper cites.
A survey of unstructured mesh generation technology
Owen, S. J · 1998
Earlier work this paper cites.
Principles of multiscale modeling
Weinan, E · 2011
Earlier work this paper cites.
U-net: Convolutional networks for biomedical image segmentation
Ronneberger, O., Fischer, P., and Brox, T · 2015
Earlier work this paper cites.
Decoupled weight decay regularization
Original
Loshchilov, I. and Hutter, F · 2017
Earlier work this paper cites.
Generating long sequences with sparse transformers
Original
Child, R., Gray, S., Radford, A., and Sutskever, I · 2019
Earlier work this paper cites.
Ccnet: Criss-cross attention for semantic segmentation
Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., and Liu, W · 2019
Earlier work this paper cites.
Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators
Original
Lu, L., Jin, P., and Karniadakis, G. E · 2019
Earlier work this paper cites.
Super-convergence: Very fast training of neural networks using large learning rates
Smith, L. N. and Topin, N · 2019
Earlier work this paper cites.
Longformer: The long-document transformer
Original
Beltagy, I., Peters, M. E., and Cohan, A · 2020
Earlier work this paper cites.
Rethinking attention with performers
Original
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al · 2020
Earlier work this paper cites.

Similar

Scaling laws for neural language models
Original
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D · 2020
Cited alongside, same era.
Transformers are rnns: Fast autoregressive transformers with linear attention
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F · 2020
Cited alongside, same era.
Reformer: The efficient transformer
Original
Kitaev, N., Kaiser, Ł., and Levskaya, A · 2020
Cited alongside, same era.
Gshard: Scaling giant models with conditional computation and automatic sharding
Original
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z · 2020
Cited alongside, same era.
Fourier neural operator for parametric partial differential equations
Original

Then

Factorized fourier neural operators
Original
Tran, A., Mathews, A., Xie, L., and Ong, C. S · 2021
Later among the works it cites.
Learning the solution operator of parametric partial differential equations with physics-informed deeponets
Wang, S., Wang, H., and Perdikaris, P · 2021
Later among the works it cites.
Towards large-scale learned solvers for parametric pdes with model-parallel fourier neural operators
Original
Grady II, T. J., Khan, R., Louboutin, M., Yin, Z., Witte, P. A., Chandra, R., Hewett, R. J., and Herrmann, F. J · 2022
Later among the works it cites.
Physics-informed machine learning: A survey on problems, methods and applications
Original
Hao, Z., Liu, S., Zhang, Y., Ying, C., Feng, Y., Su, H., and Zhu, J · 2022
Later among the works it cites.
Augmented physics-informed neural networks (apinns): A gating network-based soft domain decomposition methodology

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., and Anandkumar, A · 2020

Cited alongside, same era.

Efficient transformers: A survey

Tay, Y., Dehghani, M., Bahri, D., and Metzler, D · 2020

Cited alongside, same era.

Choose a transformer: Fourier or galerkin

Cao, S · 2021

Cited alongside, same era.

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021

Fedus, W., Zoph, B., and Shazeer, N · 2021

Cited alongside, same era.

Multiwavelet-based operator learning for differential equations

Gupta, G., Xiao, X., and Bogdan, P · 2021

Cited alongside, same era.

Extended physics-informed neural networks (xpinns): A generalized space-time domain decomposition based deep learning framework for nonlinear partial differential equations

Jagtap, A. D. and Karniadakis, G. E · 2021

Cited alongside, same era.

Swin transformer: Hierarchical vision transformer using shifted windows

Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B · 2021

Cited alongside, same era.

Random feature attention

Peng, H., Pappas, N., Yogatama, D., Schwartz, R., Smith, N. A., and Kong, L · 2021

Cited alongside, same era.

Hu, Z., Jagtap, A. D., Karniadakis, G. E., and Kawaguchi, K · 2022

Later among the works it cites.

Mionet: Learning multiple-input operators via tensor product

Jin, P., Meng, S., and Lu, L · 2022

Later among the works it cites.

Mesh-independent operator learning for partial differential equations

Lee, S · 2022

Later among the works it cites.

Ht-net: Hierarchical transformer based operator learning model for multiscale pdes

Liu, X., Xu, B., and Zhang, L · 2022

Later among the works it cites.

Variable-input deep operator networks

Prasthofer, M., De Ryck, T., and Mishra, S · 2022

Later among the works it cites.

Improved architectures and training algorithms for deep operator networks

Wang, S., Wang, H., and Perdikaris, P · 2022

Later among the works it cites.

U-fno—an enhanced fourier neural operator-based deep-learning model for multiphase flow

Wen, G., Li, Z., Azizzadenesheli, K., Anandkumar, A., and Benson, S. M · 2022

Later among the works it cites.

Nuno: A general framework for learning parametric pdes with non-uniform data

Liu, S., Hao, Z., Ying, C., Su, H., Cheng, Z., and Zhu, J · 2023

Closest in time.