Deep convolutional Ritz method: parametric PDE surrogates without labeled data

doi:10.1007/s10483-023-2992-6

摘要/Abstract

摘要： The parametric surrogate models for partial differential equations (PDEs) are a necessary component for many applications in computational sciences, and the convolutional neural networks (CNNs) have proven to be an excellent tool to generate these surrogates when parametric fields are present. CNNs are commonly trained on labeled data based on one-to-one sets of parameter-input and PDE-output fields. Recently, residual-based deep convolutional physics-informed neural network (DCPINN) solvers for parametric PDEs have been proposed to build surrogates without the need for labeled data. These allow for the generation of surrogates without an expensive offline-phase. In this work, we present an alternative formulation termed deep convolutional Ritz method (DCRM) as a parametric PDE solver. The approach is based on the minimization of energy functionals, which lowers the order of the differential operators compared to residual-based methods. Based on studies involving the Poisson equation with a spatially parameterized source term and boundary conditions, we find that CNNs trained on labeled data outperform DCPINNs in convergence speed and generalization abilities. The surrogates generated from the DCRM, however, converge significantly faster than their DCPINN counterparts, and prove to generalize faster and better than the surrogates obtained from both CNNs trained on labeled data and DCPINNs. This hints that the DCRM could make PDE solution surrogates trained without labeled data possibly.

关键词: physics-informed constraint, physics-informed neural network (PINN), deep energy network, convolutional neural network (CNN)

Abstract: The parametric surrogate models for partial differential equations (PDEs) are a necessary component for many applications in computational sciences, and the convolutional neural networks (CNNs) have proven to be an excellent tool to generate these surrogates when parametric fields are present. CNNs are commonly trained on labeled data based on one-to-one sets of parameter-input and PDE-output fields. Recently, residual-based deep convolutional physics-informed neural network (DCPINN) solvers for parametric PDEs have been proposed to build surrogates without the need for labeled data. These allow for the generation of surrogates without an expensive offline-phase. In this work, we present an alternative formulation termed deep convolutional Ritz method (DCRM) as a parametric PDE solver. The approach is based on the minimization of energy functionals, which lowers the order of the differential operators compared to residual-based methods. Based on studies involving the Poisson equation with a spatially parameterized source term and boundary conditions, we find that CNNs trained on labeled data outperform DCPINNs in convergence speed and generalization abilities. The surrogates generated from the DCRM, however, converge significantly faster than their DCPINN counterparts, and prove to generalize faster and better than the surrogates obtained from both CNNs trained on labeled data and DCPINNs. This hints that the DCRM could make PDE solution surrogates trained without labeled data possibly.

Key words: physics-informed constraint, physics-informed neural network (PINN), deep energy network, convolutional neural network (CNN)

中图分类号:

35-02
92B20

J. N. FUHG, A. KARMARKAR, T. KADEETHUM, H. YOON, N. BOUKLAS. Deep convolutional Ritz method: parametric PDE surrogates without labeled data[J]. Applied Mathematics and Mechanics (English Edition), 2023, 44(7): 1151-1174.

参考文献

[1] GOGU, C. Improving the efficiency of large scale topology optimization through on-the-fly reduced order model construction. International Journal for Numerical Methods in Engineering, 101(4), 281-304(2015)
[2] XIA, L. and BREITKOPF, P. A reduced multiscale model for nonlinear structural topology optimization. Computer Methods in Applied Mechanics and Engineering, 280, 117-134(2014)
[3] KESHAVARZZADEH, V., KIRBY, R. M., and NARAYAN, A. Robust topology optimization with low rank approximation using artificial neural networks. Computational Mechanics, 68(6), 1297-1323(2021)
[4] ROACHE, P. J. Quantification of uncertainty in computational fluid dynamics. Annual Review of Fluid Mechanics, 29(1), 123-160(1997)
[5] CHEN, P., QUARTERONI, A., and ROZZA, G. Reduced basis methods for uncertainty quantification. SIAM/ASA Journal on Uncertainty Quantification, 5(1), 813-869(2017)
[6] TRIPATHY, R. K. and BILIONIS, I. Deep UQ:learning deep neural network surrogate models for high dimensional uncertainty quantification. Journal of Computational Physics, 375, 565-588(2018)
[7] BIEGLER, L. T., GHATTAS, O., HEINKENSCHLOSS, M., and WAAMDERS, B. V. B. Large-scale PDE-constrained optimization:an introduction. Real-Time PDE-Constrained Optimization, Springer, Berlin/Heidelberg (2003)
[8] FAHL, M. and SACHS, E. W. Reduced order modelling approaches to PDE-constrained optimization based on proper orthogonal decomposition. Large-Scale PDE-Constrained Optimization, Springer, Berlin/Heidelberg (2003)
[9] ZAHR, M. J. and FARHAT, C. Progressive construction of a parametric reduced-order model for PDE-constrained optimization. International Journal for Numerical Methods in Engineering, 102(5), 1111-1135(2015)
[10] FUHG, J. N., BOEHM, C., BOUKLAS, N., FAU, A., WRIGGERS, P., and MARINO, M. Model-data-driven constitutive responses:application to a multiscale computational framework. International Journal of Engineering Science, 167, 103522(2021)
[11] FUHG, J. N., MARINO, M., and BOUKLAS, N. Local approximate Gaussian process regression for data-driven constitutive models:development and comparison with neural networks. Computer Methods in Applied Mechanics and Engineering, 388, 114217(2022)
[12] FUHG, J. N. and BOUKLAS, N. On physics-informed data-driven isotropic and anisotropic constitutive models through probabilistic machine learning and space-filling sampling. Computer Methods in Applied Mechanics and Engineering, 394, 114915(2022)
[13] WRIGGERS, P. Nonlinear Finite Element Methods, Springer, Berlin/Heidelberg (2008)
[14] MOUKALLED, F., MANGANI, L., and DARWISH, M. The finite volume method. The Finite Volume Method in Computational Fluid Dynamics, Springer, Berlin/Heidelberg (2016)
[15] BERKOOZ, G., HOLMES, A. P., and LUMLEY, J. L. The proper orthogonal decomposition in the analysis of turbulent flows. Annual Review of Fluid Mechanics, 25(1), 539-575(1993)
[16] COUPLET, M., BASDEVANT, C., and SAGAUT, P. Calibrated reduced-order POD-Galerkin system for fluid flow modelling. Journal of Computational Physics, 207(1), 192-220(2005)
[17] GUO, M. W. and HESTHAVEN, J. S. Reduced order modeling for nonlinear structural analysis using Gaussian process regression. Computer Methods in Applied Mechanics and Engineering, 341, 807-826(2018)
[18] ORTALI, G., DEMO, N., and ROZZA, G. Gaussian process approach within a data-driven POD framework for fluid dynamics engineering problems. arXiv Preprint, arXiv:2012.01989(2020) https://doi.org/10.48550/arXiv.2012.01989
[19] BHATTACHARYA, K., HOSSEINI, B., KOVACHKI, N. B., and STUART, A. M. Model reduction and neural networks for parametric PDEs. arXiv Preprint, arXiv:2005.03180(2020) https://doi.org/10.48550/arXiv.2005.03180
[20] ZHU, Y. H. and ZABARAS, N. Bayesian deep convolutional encoder-decoder networks for surrogate modeling and uncertainty quantification. Journal of Computational Physics, 366, 415-447(2018)
[21] KUTYNIOK, G., PETERSEN, P., RASLAN, M., and SCHNEIDER, R. A theoretical analysis of deep neural networks and parametric PDEs. Constructive Approximation, 55(1), 73-125(2022)
[22] KHOO, Y. H., LU, J. F., and YING, L. X. Solving parametric PDE problems with artificial neural networks. European Journal of Applied Mathematics, 32(3), 421-435(2021)
[23] SUN, A. Y. Discovering state-parameter mappings in subsurface models using generative adversarial networks. Geophysical Research Letters, 45(20), 11137-11146(2018)
[24] KADEETHUM, T., O'MALLEY, D., FUHG, J. K., CHOI, Y., LEE, J., VISWANATHAN, H. S., and BOUKLAS, N. A framework for data-driven solution and parameter estimation of PDEs using conditional generative adversarial networks. Nature Computational Science, 1(12), 819-829(2021)
[25] LI, Z. Y., KOVACHKI, N., AZIZZADENESHELI, K., LIU, B., BHATTACHARYA, K., STUART, A., and ANANDKUMAR, A. Neural operator:graph kernel network for partial differential equations. arXiv Preprint, arXiv:2003.03485(2020) https://doi.org/10.48550/arXiv.2003.03485
[26] LI, Z. Y., KOVACHKI, N. B., AZIZZADENESHELI, K., LIU, B., BHATTACHARYA, K., STUART, A., and ANANDKUMAR, A. Fourier neural operator for parametric partial differential equations. arXiv Preprint, arXiv:2010.08895(2020) https://arxiv.org/abs/2010.08895
[27] LIU, H. T., ONG, Y. S., and CAI, J. F. A survey of adaptive sampling for global metamodeling in support of simulation-based complex engineering design. Structural and Multidisciplinary Optimization, 57(1), 393-416(2018)
[28] FUHG, J. N., FAU, A., and NACKENHORST, U. State-of-the-art and comparative review of adaptive sampling methods for kriging. Archives of Computational Methods in Engineering, 28(4), 2689-2747(2021)
[29] FUHG, J. N. and FAU, A. A classification-pursuing adaptive approach for Gaussian process regression on unlabeled data. Mechanical Systems and Signal Processing, 162, 107976(2022)
[30] SCHOBI, R., SUDRET, B., and WIART, J. Polynomial-chaos-based kriging. arXiv Preprint, arXiv:1502.03939(2015) https://doi.org/10.48550/arXiv.1502.03939
[31] WANG, R., KASHINATH, K., MUSTAFA, M., ALBERT, A., and YU, R. Towards physics-informed deep learning for turbulent flow prediction. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery&Data Mining, Virtual Event, Association for Computing Machinery, 1457-1466(2020)
[32] MOHAN, A. T., LUBBERS, N., LIVESCU, D., and CHERTKOV, M. Embedding hard physical constraints in neural network coarse-graining of 3D turbulence. arXiv Preprint, arXiv:2002.00021(2020) https://doi.org/10.48550/arXiv.2002.00021
[33] GRIEWANK, A. On automatic differentiation. Mathematical Programming:Recent Developments and Applications, Kluwer Academic Publishers, The Netherlands (1989)
[34] LAGARIS, I. E., LIKAS, A., and FOTIADIS, D. I. Artificial neural networks for solving ordinary and partial differential equations. IEEE Transactions on Neural Networks, 9(5), 987-1000(1998)
[35] RAISSI, M., PERDIKARIS, P., and KARNIADAKIS, G. E. Physics-informed neural networks:a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, 686-707(2019)
[36] WESSELS, H., WEIβENFELS, C., and WRIGGERS, P. The neural particle method——an updated lagrangian physics informed neural network for computational fluid dynamics. Computer Methods in Applied Mechanics and Engineering, 368, 113127(2020)
[37] FUHG, J. N., KALOGERIS, I., FAU, A., and BOUKLAS, N. Interval and fuzzy physics-informed neural networks for uncertain fields. Probabilistic Engineering Mechanics, 68, 103240(2022)
[38] LU, L., JIN, P. Z., and KARNIADAKIS, G. E. Deeponet:learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv Preprint, arXiv:1910.03193(2019) https://doi.org/10.48550/arXiv.1910.03193
[39] LU, L., JIN, P. Z., PANG, G. F., ZHANG, Z. Q., and KARNIADAKIS, G. E. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3), 218-229(2021)
[40] WANG, S. F., WANG, H. W., and PERDIKARIS, P. Learning the solution operator of parametric partial differential equations with physics-informed DeepONets. Science Advances, 7(40), eabi8605(2021)
[41] ZHU, Y. H., ZABARAS, N., KOUTSOURELAKIS, P. S., and PERDIKARIS, P. Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data. Journal of Computational Physics, 394, 56-81(2019)
[42] GAO, H., SUN, L. N., and WANG, J. X. PhyGeoNet:physics-informed geometry-adaptive convolutional neural networks for solving parameterized steady-state PDEs on irregular domain. Journal of Computational Physics, 428, 110079(2021)
[43] REN, P., RAO, C. Q., LIU, Y., WANG, J. X., and SUN, H. PhyCRNet:physics-informed convolutional-recurrent network for solving spatiotemporal PDEs. Computer Methods in Applied Mechanics and Engineering, 389, 114399(2022)
[44] WEINAN, E. and YU, B. The deep Ritz method:a deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1), 1-12(2018)
[45] LIAO, Y. L. and MING, P. B. Deep Nitsche method:deep Ritz method with essential boundary conditions. Communications in Computational Physics, 29, 1365-1384(2021)
[46] DUAN, C. G., JIAO, Y. L., LAI, Y. M., LU, X. L., and YANG, Z. J. Convergence rate analysis for deep Ritz method. Communications in Computational Physics, 31(4), 1020-1048(2022)
[47] SAMANIEGO, E., ANITESCU, C., GOSWAMI, S., NGUYEN-THANH, V. M., GUO, H. W., HAMDIA, K., ZHUANG, X., and RABCZUK, T. An energy approach to the solution of partial differential equations in computational mechanics via machine learning:concepts, implementation and applications. Computer Methods in Applied Mechanics and Engineering, 362, 112790(2020)
[48] FUHG, J. N. and BOUKLAS, N. The mixed deep energy method for resolving concentration features in finite strain hyperelasticity. Journal of Computational Physics, 451, 110839(2022)
[49] KRISHNAPRIYAN, A., GHOLAMI, A., ZHE, S. D., KIRBY, R., and MAHONEY, M. W. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems, 34, 26548-26560(2021)
[50] WANG, S. F., YU, X. L., and PERDIKARIS, P. When and why pinns fail to train:a neural tangent kernel perspective. Journal of Computational Physics, 449, 110768(2022)
[51] ANDERSON, I. and DUCHAMP, T. On the existence of global variational principles. American Journal of Mathematics, 102(5), 781-868(1980)
[52] REDDY, J. N. Energy Principles and Variational Methods in Applied Mechanics, John Wiley&Sons, New York (2017)
[53] BEIRO, D. A., VEIGA, L., BREZZI, F., MARINI, L. D., and RUSSO, A. The hitchhiker's guide to the virtual element method. Mathematical Models and Methods in Applied Sciences, 24(8), 1541-1573(2014)
[54] DOUGLAS, J. Solution of the inverse problem of the calculus of variations. Proceedings of the National Academy of Sciences, 25(12), 631-637(1940)
[55] TAKENS, F. A global version of the inverse problem of the calculus of variations. Journal of Differential Geometry, 14(4), 543-562(1979)
[56] ZENKOV, D. V. The Inverse Problem of the Calculus of Variations:Local and Global Theory and Applications, Atlantis Press, North Carolina (2015)
[57] WEINSTOCK, R. Calculus of Variations:with Applications to Physics and Engineering, Dover Publications, New York (1974)
[58] EVANS, L. C. Partial Differential Equations, American Mathematical Society, Washington, D.C.(1998)
[59] ALGUACIL, A., PINTO,W. G., BAUERHEIM, M., JACOB, M. C., and MOREAU, S. Effects of boundary conditions in fully convolutional networks for learning spatiotemporal dynamics. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer, The Netherlands, 102-117(2021)
[60] MASCI, J., BOSCAINI, D., BRONSTEIN, M. M., and VANDERGHEYNST, P. Geodesic convolutional neural networks on Riemannian manifolds. IEEE International Conference on Computer Vision Workshops, IEEE, 37-45(2015)
[61] QI, C. R., LI, Y., HAO, S., and GUIBAS, L. J. Pointnet++:deep hierarchical feature learning on point sets in a metric space. Proceedings of the$31$st International Conference on Neural Information Processing Systems, Perth, 5105-5114(2017)
[62] JIANG, C. Y., WANG, D. Q., HUANG, J. W., MARCUS, P., and NIEβNER, M. Convolutional neural networks on non-uniform geometrical signals using Euclidean spectral transformation. International Conference on Learning Representations-2019, International Society for Machine Learning, New Orleans (2019)
[63] GU, J. X., WANG, Z. H., KUEN, J., MA, L. Y., SHAHROUDY, A., SHUAI, B., LIU, T., WANG, X. X., WANG, L., WANG, G., CAI, J. F., and CHEN, T. Recent advances in convolutional neural networks. Pattern Recognition, 77, 354-377(2018)
[64] ESTRACH, J. B., SZLAM, A., and LECUN, Y. Signal recovery from pooling representations. Proceedings of Machine Learning Research, 32(2), 307-315(2014)
[65] HINTON, G. E., SRIVASTAVA, N., KRIZHEVSKY, A., SUTSKEVER, I., and SALAKHUTDINOV, R. R. Improving neural networks by preventing co-adaptation of feature detectors. arXiv Preprint, arXiv:1207.0580(2012) https://doi.org/10.48550/arXiv.1207.0580
[66] SRIVASTAVA, N., HINTON, G., KRIZHEVSKY, A., SUTSKEVER, I., and SALAKHUTDINOV, R. Dropout:a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1), 1929-1958(2014)
[67] IOFFE, S. and SZEGEDY, C. Batch normalization:accelerating deep network training by reducing internal covariate shift. arXiv Preprint, arXiv:1502.03167(2015) https://doi.org/10.48550/arXiv.1502.03167
[68] RONNEBERGER, O., FISCHER, P., and BROX, T. U-net:convolutional networks for biomedical image segmentation. arXiv Preprint, arXiv:1505.04597(2015) https://doi.org/10.48550/arXiv.1505.04597
[69] MAO, X. J., SHEN, C. H., and YANG, Y. B. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections.$30$th Conference on Neural Information Processing Systems, the Neural Information Processing Systems (NIPS) Foundation, Barcelona (2016)
[70] WANG, W., HUANG, Y., WANG, Y. Z., and WANG, L. Generalized autoencoder:a neural network framework for dimensionality reduction. 27th IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Columbus (2014)
[71] BADRINARAYANAN, V., KENDALL, A., and CIPOLLA, R. SegNet:a deep convolutional encoderdecoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12), 2481-2495(2017)
[72] HINTON, G. E. and SALAKHUTDINOV, R. R. Reducing the dimensionality of data with neural networks. Science, 313(5786), 504-507(2006)
[73] ROWEIS, S. and BRODY, C. Linear Heteroencoders, Technical report, Gatsby Computational Neuroscience Unit, Alexandra House, London (1999)
[74] BRIDGMAN, W., ZHANG, X., TEICHERT, G., KHALIL, M., GARIKIPATI, K., and JONES, R. A heteroencoder architecture for prediction of failure locations in porous metals using variational inference. Computer Methods in Applied Mechanics and Engineering, 398, 115236(2022)
[75] KINGMA, D. and BA, J. Adam:a method for stochastic optimization. arXiv Preprint, arXiv:1412.6980(2014) https://doi.org/10.48550/arXiv.1412.6980
[76] MONTGOMERY, D. C., PECK, E. A., and VINING, G. G. Introduction to Linear Regression Analysis, John Wiley&Sons, New York (2021)
[77] SEBER, G. A. F. and WILD, C. J. Nonlinear Regression, John Wiley&Sons, New York (1990)
[78] LEVEQUE, R. J. Finite Difference Methods for Ordinary and Partial Differential Equations:Steady-State and Time-Dependent Problems, Society for Industrial and Applied Mathematics, Philadelphia (2007)
[79] HAMEL, C. M., LONG, K. N., and KRAMER, S. L. Calibrating constitutive models with full-field data via physics informed neural networks. arXiv Preprint, arXiv:2203.16577(2022) https://doi.org/10.48550/arXiv.2203.16577
[80] RITZ, W.Über eine neue Methode zur Lösung Gewisser Variationsprobleme der Mathematischen Physik, Walter de Gruyter, New York (1909)
[81] LEISSA, A. W. The historical bases of the Rayleigh and Ritz methods. Journal of Sound and Vibration, 287(4-5), 961-978(2005)
[82] DAVIS, P. J. and RABINOWITZ, P. Methods of Numerical Integration, Dover Publications, New York (2007)
[83] PASZKE, A., GROSS, S., MASSA, F., ADAM LERER, A., BRADBURY, J., CHANAN, G., KILLEEN, T., LIN, Z., GIMELSHEIN, N., ANTIGA, L., DESMAISON, A., KÖPF, A., YANG, E., DEVITO, Z., RAISON, M., TEJANI, A., CHILAMKURTHY, S., STEINER, B., FANG, L., BAI, J., and CHINTALA, S. Pytorch:an imperative style, high-performance deep learning library. arXiv Preprint, arXiv:1912.01703(2019) https://doi.org/10.48550/arXiv.1912.01703
[84] BAI, J. S., RABCZUK, T., GUPTA, A., ALZUBAIDI, L., and GU, Y. T. A physics-informed neural network technique based on a modified loss function for computational 2D and 3D solid mechanics. Computational Mechanics, 71, 543-562(2023)
[85] SHI, X. J., CHEN, Z. R., WANG, H., YEUNG, D. Y., WONG, W. K., and WOO, W. C. Convolutional LSTM network:a machine learning approach for precipitation nowcasting. Advances in Neural Information Processing Systems, 28, 802-810(2015)