Variational inference in neural functional prior using normalizing flows: application to differential equation and operator learning problems

doi:10.1007/s10483-023-2997-7

摘要/Abstract

摘要： Physics-informed deep learning has recently emerged as an effective tool for leveraging both observational data and available physical laws. Physics-informed neural networks (PINNs) and deep operator networks (DeepONets) are two such models. The former encodes the physical laws via the automatic differentiation, while the latter learns the hidden physics from data. Generally, the noisy and limited observational data as well as the over-parameterization in neural networks (NNs) result in uncertainty in predictions from deep learning models. In paper "MENG, X., YANG, L., MAO, Z., FERRANDIS, J. D., and KARNIADAKIS, G. E. Learning functional priors and posteriors from data and physics. Journal of Computational Physics, 457, 111073 (2022)" has two stages: (i) prior learning, and (ii) posterior estimation. At the first stage, the GANs are utilized to learn a functional prior either from a prescribed function distribution, e.g., the Gaussian process, or from historical data and available physics. At the second stage, the Hamiltonian Monte Carlo (HMC) method is utilized to estimate the posterior in the latent space of GANs. However, the vanilla HMC does not support the mini-batch training, which limits its applications in problems with big data. In the present work, we propose to use the normalizing flow (NF) models in the context of variational inference (VI), which naturally enables the mini-batch training, as the alternative to HMC for posterior estimation in the latent space of GANs. A series of numerical experiments, including a nonlinear differential equation problem and a 100-dimensional (100D) Darcy problem, are conducted to demonstrate that the NFs with full-/mini-batch training are able to achieve similar accuracy as the "gold rule" HMC. Moreover, the mini-batch training of NF makes it a promising tool for quantifying uncertainty in solving the high-dimensional partial differential equation (PDE) problems with big data.

关键词: uncertainty quantification (UQ), physics-informed neural network (PINN), deep operator network (DeepONet), generative adversarial network (GAN), normalizing flow (NF), differential equation

Abstract: Physics-informed deep learning has recently emerged as an effective tool for leveraging both observational data and available physical laws. Physics-informed neural networks (PINNs) and deep operator networks (DeepONets) are two such models. The former encodes the physical laws via the automatic differentiation, while the latter learns the hidden physics from data. Generally, the noisy and limited observational data as well as the over-parameterization in neural networks (NNs) result in uncertainty in predictions from deep learning models. In paper "MENG, X., YANG, L., MAO, Z., FERRANDIS, J. D., and KARNIADAKIS, G. E. Learning functional priors and posteriors from data and physics. Journal of Computational Physics, 457, 111073 (2022)" has two stages: (i) prior learning, and (ii) posterior estimation. At the first stage, the GANs are utilized to learn a functional prior either from a prescribed function distribution, e.g., the Gaussian process, or from historical data and available physics. At the second stage, the Hamiltonian Monte Carlo (HMC) method is utilized to estimate the posterior in the latent space of GANs. However, the vanilla HMC does not support the mini-batch training, which limits its applications in problems with big data. In the present work, we propose to use the normalizing flow (NF) models in the context of variational inference (VI), which naturally enables the mini-batch training, as the alternative to HMC for posterior estimation in the latent space of GANs. A series of numerical experiments, including a nonlinear differential equation problem and a 100-dimensional (100D) Darcy problem, are conducted to demonstrate that the NFs with full-/mini-batch training are able to achieve similar accuracy as the "gold rule" HMC. Moreover, the mini-batch training of NF makes it a promising tool for quantifying uncertainty in solving the high-dimensional partial differential equation (PDE) problems with big data.

Key words: uncertainty quantification (UQ), physics-informed neural network (PINN), deep operator network (DeepONet), generative adversarial network (GAN), normalizing flow (NF), differential equation

中图分类号:

76F25

Xuhui MENG. Variational inference in neural functional prior using normalizing flows: application to differential equation and operator learning problems[J]. Applied Mathematics and Mechanics (English Edition), 2023, 44(7): 1111-1124.

参考文献

[1] MENG, X., YANG, L., MAO, Z., FERRANDIS, J. D., and KAENIADAKIS, G. E. Learning functional priors and posteriors from data and physics. Journal of Computational Physics, 457, 111073(2022)
[2] MENG, X. and KARNIADAKIS, G. E. A composite neural network that learns from multi fidelity data:application to function approximation and inverse PDE problems. Journal of Computational Physics, 401, 109020(2020)
[3] MENG, X., WANG, Z., FAN, D., TRIANTAFYLLOU, M. S., and KARNIADAKIS, G. E. A fast multi-fidelity method with uncertainty quantification for complex data correlations:application to vortex-induced vibrations of marine risers. Computer Methods in Applied Mechanics and Engineering, 386, 114212(2021)
[4] MENG, X., BABAEE, H., and KARNIADAKIS, G. E. Multi-fidelity bayesian neural networks:algorithms and applications. Journal of Computational Physics, 438, 110361(2021)
[5] RAISSI, M., PERDIKARIS, P., and KARNIADAKIS, G. E. Physics-informed neural networks:a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, 686-707(2019)
[6] SIRIGNANO, J. and SPILIOPOULOS, K. DGM:a deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375, 1339-1364(2018)
[7] HAN, J. and JENTZEN, A. W. E. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34), 8505-8510(2018)
[8] WEINAN, E. and YU, B. The deep Ritz method:a deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1), 1-12(2018)
[9] LU, L., JIN, P., PANG, G., ZHANG, Z., and KARNIADAKIS, G. E. Learning nonlinear opera-tors via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3), 218-229(2021)
[10] LI, Z., KOVACHKI, N., AZIZZADENESHELI, K., LIU, B., BHATTACHARYA, K., STUART, A., and ANANDKUMAR, A. Fourier neural operator for parametric partial differential equations. arXiv Preprint, arXiv:2010.08895(2020) https://doi.org/10.48550/arXiv.2010.08895
[11] ABDAR, M., POURPANAH, F., HUSSAIN, S., REZAZADEGAN, D., LIU, L., GHAVAMZADEH, M., FIEGUTH, P., CAO, X. C., KHOSRAVI, A., ACHARYA, U. R., MAKARENKOV, V., and NAHAVANDI, S. A review of uncertainty quantification in deep learning:techniques, applications and challenges. Information Fusion, 76, 243-297(2021)
[12] PICKERING, E., GUTH, S., KARNIADAKIS, G. E., and SAPSIS, T. P. Discovering and forecasting extreme events via active learning in neural operators. Nature Computational Science, 2(12), 823-833(2022)
[13] LINKA, K., SCHÄFER, A., MENG, X., ZOU, Z., KARNIADAKIS, G. E., and KUHL, E. Bayesian physics informed neural networks for real-world nonlinear dynamical systems. Computer Methods in Applied Mechanics and Engineering, 402, 115346(2022)
[14] NEAL, R. M. Bayesian Learning for Neural Networks, Springer Science and Business Media, Berlin (2012)
[15] YANG, L., MENG, X., and KARNIADAKIS, G. E. B-PINNs:Bayesian physics-informed neural networks for forward and inverse PDE problems with noisy data. Journal of Computational Physics, 425, 109913(2021)
[16] LAKSHMINARAYANAN, B., PRITZEL, A., and BLUNDELL, C. Simple and scalable predictive uncertainty estimation using deep ensembles. 31st Annual Conference on Neural Information Processing Systems, Long Beach, State of California (2017)
[17] PEARCE, T., LEIBFRIED, F., and BRINTRUP, A. Uncertainty in neural networks:approx-imately Bayesian ensembling. International Conference on Artificial Intelligence and Statistics, 234-244(2020)
[18] ZHANG, D., LU, L., GUO, L., and KARNIADAKIS, G. E. Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems. Journal of Computational Physics, 397, 108850(2019)
[19] YAO, J., PAN, W., GHOSH, S., and DOSHI-VELEZ, F. Quality of uncertainty quantificatio for Bayesian neural network inference. arXiv Preprint, arXiv:1906.09686(2019) https://doi.org/10.48550/arXiv.1906.09686
[20] PSAROS, A. F., MENG, X., ZOU, Z., GUO, L., and KARNIADAKIS, G. E. Uncertainty quantification in scientific machine learning:methods, metrics, and comparisons. Journal of Computational Physics, 477, 111902(2023)
[21] CHEN, T., FOX, E., and GUESTRIN, C. Stochastic gradient Hamiltonian Monte Carlo. International Conference on Machine Learning, 1683-1691(2014)
[22] BLUNDELL, C., CORNEBISE, J., KAVUKCUOGLU, K., and WIERSTRA, D. Weight uncertainty in neural networks. arXiv Preprint, arXiv:1505.05424(2015) https://doi.org/10.48550/arXiv.1505.05424
[23] REZENDE, D. and MOHAMED, S. Variational inference with normalizing flows. International Conference on Machine Learning, 1530-1538(2015)
[24] ZOU, Z., MENG, X., PSAROS, A. F., and KARNIADAKIS, G. E. NeuralUQ:a comprehensive library for uncertainty quantification in neural differential equations and operators. arXiv Preprint, arXiv:2208.11866(2022) https://doi.org/10.48550/arXiv.2208.11866
[25] DINH, L., SOHL-DICKSTEIN, J., and BENGIO, S. Density estimation using real NVP. arXiv Preprint, arXiv:1605.08803(2016) https://doi.org/10.48550/arXiv.1605.08803
[26] PAPAMAKARIOS, G., PAVLAKOU, T., and MURRAY, I. Masked autoregressive flow for density estimation. 31st Annual Conference on Neural Information Processing Systems, Long Beach, State of California (2017)
[27] KINGMA, D. P., SALIMANS, T., JOZEFOWICZ, R., CHEN, X., SUTSKEVER, I., and WELLING, M. Improved variational inference with inverse autoregressive flow. 30th Conference on Neural Information Processing Systems, Barcelona, Spain (2016)
[28] KINGMA, D. P. and DHARIWAL, P. Glow:generative flow with invertible 1×1 convolutions. 31st Annual Conference on Neural Information Processing Systems, Long Beach, State of California (2017)
[29] LU, L., MENG, X., CAI, S., MAO, Z., GOSWAMI, S., ZHANG, Z., and KARNIADAKIS, G. E. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineering, 393, 114778(2022)
[30] HOFFMAN, M. D. and GELMAN, A. The No-U-Turn sampler:adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15(1), 1593-1623(2014)
[31] LAO, J., SUTER, C., LANGMORE, I., CHIMISOV, C., SAXENA, A., SOUNTSOV, P., MOORE, D., SAUROUS, R. A., HOFFMAN, M. D., and DILLON, J. V. Tfp. mcmc:modern Markov chain Monte Carlo tools built for modern hardware. arXiv Preprint, arXiv:2002.01184 https://doi.org/10.48550/arXiv.2002.01184