A dive into spectral inference networks: improved algorithms for self-supervised learning of continuous spectral representations

doi:10.1007/s10483-023-2998-7

Applied Mathematics and Mechanics (English Edition) ›› 2023, Vol. 44 ›› Issue (7): 1199-1224.doi: https://doi.org/10.1007/s10483-023-2998-7

• 论文 • 上一篇

A dive into spectral inference networks: improved algorithms for self-supervised learning of continuous spectral representations

J. WU¹, S. F. WANG¹, P. PERDIKARIS²

1. Graduate Group in Applied Mathematics and Computational Science, University of Pennsylvania Philadelphia, PA 19104, U.S.A.;
2. Department of Mechanical Engineering and Applied Mechanics, University of Pennsylvania Philadelphia, PA 19104, U.S.A.

收稿日期:2023-02-21 修回日期:2023-06-02 出版日期:2023-07-01 发布日期:2023-07-05
通讯作者: P. PERDIKARIS, E-mail:pgp@seas.upenn.edu
基金资助:
the U.S. Department of Energy under the Advanced Scientific Computing Research Program (No. DE-SC0019116) and the U.S. Air Force Office of Scientific Research (No. AFOSR FA9550-20-1-0060)

A dive into spectral inference networks: improved algorithms for self-supervised learning of continuous spectral representations

J. WU¹, S. F. WANG¹, P. PERDIKARIS²

1. Graduate Group in Applied Mathematics and Computational Science, University of Pennsylvania Philadelphia, PA 19104, U.S.A.;
2. Department of Mechanical Engineering and Applied Mechanics, University of Pennsylvania Philadelphia, PA 19104, U.S.A.

Received:2023-02-21 Revised:2023-06-02 Online:2023-07-01 Published:2023-07-05
Contact: P. PERDIKARIS, E-mail:pgp@seas.upenn.edu
Supported by:
the U.S. Department of Energy under the Advanced Scientific Computing Research Program (No. DE-SC0019116) and the U.S. Air Force Office of Scientific Research (No. AFOSR FA9550-20-1-0060)

摘要/Abstract

摘要： We propose a self-supervising learning framework for finding the dominant eigenfunction-eigenvalue pairs of linear and self-adjoint operators. We represent target eigenfunctions with coordinate-based neural networks and employ the Fourier positional encodings to enable the approximation of high-frequency modes. We formulate a self-supervised training objective for spectral learning and propose a novel regularization mechanism to ensure that the network finds the exact eigenfunctions instead of a space spanned by the eigenfunctions. Furthermore, we investigate the effect of weight normalization as a mechanism to alleviate the risk of recovering linear dependent modes, allowing us to accurately recover a large number of eigenpairs. The effectiveness of our methods is demonstrated across a collection of representative benchmarks including both local and non-local diffusion operators, as well as high-dimensional time-series data from a video sequence. Our results indicate that the present algorithm can outperform competing approaches in terms of both approximation accuracy and computational cost.

关键词: spectral learning, partial differential equation (PDE), neural network, slow features analysis

Abstract: We propose a self-supervising learning framework for finding the dominant eigenfunction-eigenvalue pairs of linear and self-adjoint operators. We represent target eigenfunctions with coordinate-based neural networks and employ the Fourier positional encodings to enable the approximation of high-frequency modes. We formulate a self-supervised training objective for spectral learning and propose a novel regularization mechanism to ensure that the network finds the exact eigenfunctions instead of a space spanned by the eigenfunctions. Furthermore, we investigate the effect of weight normalization as a mechanism to alleviate the risk of recovering linear dependent modes, allowing us to accurately recover a large number of eigenpairs. The effectiveness of our methods is demonstrated across a collection of representative benchmarks including both local and non-local diffusion operators, as well as high-dimensional time-series data from a video sequence. Our results indicate that the present algorithm can outperform competing approaches in terms of both approximation accuracy and computational cost.

Key words: spectral learning, partial differential equation (PDE), neural network, slow features analysis

中图分类号:

65D15

J. WU, S. F. WANG, P. PERDIKARIS. A dive into spectral inference networks: improved algorithms for self-supervised learning of continuous spectral representations[J]. Applied Mathematics and Mechanics (English Edition), 2023, 44(7): 1199-1224.

参考文献

[1] HAVLICEK, J. P., HARDING, D. S., and BOVIK, A. C. Multidimensional quasi-eigenfunction approximations and multicomponent AM-FM models. IEEE Transactions on Image Processing, 9(2), 227-242(2000)
[2] GILBOA, G. Nonlinear Eigenproblems in Image Processing and Computer Vision, Springer, Berlin (2018)
[3] WOLD, S., ESBENSEN, K., and GELADI, P. Principal component analysis. Chemometrics and Intelligent Laboratory Systems, 2(1-3), 37-52(1987)
[4] BENESTY, J. Adaptive eigenvalue decomposition algorithm for passive acoustic source localization. The Journal of the Acoustical Society of America, 107(1), 384-391(2000)
[5] MARBURG, S. DIENEROWITZ, F., HORST, T., and SCHNEIDER, S. Normal modes in external acoustics, part II: eigenvalues and eigenvectors in 2D. Acta Acustica United with Acustica, 92(1), 97-111(2006)
[6] NAKA,Y., OBERAI, A. A., and SHINN-CUNNINGHAM, B. G. Acoustic eigenvalues of rectangular rooms with arbitrary wall impedances using the interval newton/ generalized bisection method. The Journal of the Acoustical Society of America, 118(6), 3662-3671(2005)
[7] THOMSON, W. T. Theory of Vibration with Applications, CRC Press, Boca Raton (2018)
[8] GLADWELL, G. M. L. Inverse Problems in Vibration, Martinus Nijhoff Publishers, Ontario (1986)
[9] MOTTERSHEAD, J. E. and RAM, Y. M. Inverse eigenvalue problems in vibration absorption: passive modification and active control. Mechanical Systems and Signal Processing, 20(1), 5-44(2006)
[10] DAYA, D. M. and POTIER-FERRY, M. A numerical method for nonlinear eigenvalue problems application to vibrations of viscoelastic structures. Computers & Structures, 79(5), 533-541(2001)
[11] SAAD, Y. Numerical Methods for Large Eigenvalue Problems: Revised Edition, Halsted Press, New York (2011)
[12] YUKAWA, T. Lax form of the quantum mechanical eigenvalue problem. Physics Letters A, 116(5), 227-230(1986)
[13] KÖVECSES, J. and FONT-LLAGUNES, J. M. An eigenvalue problem for the analysis of variable topology mechanical systems. Journal of Computational and Nonlinear Dynamics, 4(3), 031006(2009)
[14] BOSCAIN, U., GAUTHIER, J. P., ROSSI, F., and SIGALOTTI, M. Approximate controllability, exact controllability, and conical eigenvalue intersections for quantum mechanical systems. Communications in Mathematical Physics, 333(3), 1225-1239(2015)
[15] KHMELNYTSKAYA, K. V., KRAVCHENKO, V. V., and ROSU, H. C. Eigenvalue problems, spectral parameter power series, and modern applications. Mathematical Methods in the Applied Sciences, 38(10), 1945-1969(2015)
[16] BIRMAN, M. S. and SOLOMJAK, M.Z. Spectral Theory of Self-Adjoint Operators in Hilbert Space, Springer Science & Business Media, Berlin (2012)
[17] BRÜNING, J., GEYLER, V., and PANKRASHKIN, K. Spectra of self-adjoint extensions and applications to solvable Schrödinger operators. Reviews in Mathematical Physics, 20(1), 1-70(2008)
[18] MCMILLAN, W. L. Ground state of liquid he 4 in one dimension. Physical Review, 138(2), A442(1965)
[19] WEINAN, E. and YU, B. The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1), 1-12(2018)
[20] HAN, J. Q., LU, J. F., and ZHOU, M. Solving high-dimensional eigenvalue problems using deep neural networks: a diffusion Monte Carlo like approach. Journal of Computational Physics, 432, 109792(2020)
[21] PFAU, D., PETERSEN, S., AGARWAL, A., BARRETT, D. G. T., and STACHENFELD, K. L. Spectral inference networks: unifying deep and spectral learning. arXiv Preprint, arXiv: 1806.02215(2018) https://doi.org/10.48550/arXiv.1806.02215
[22] CHOO, K., MEZZACAPO, A., and CARLEO, G. Fermionic neural-network states for ab-initio electronic structure. Nature Communications, 11(1), 2368(2020)
[23] BEN-SHAUL, I., BAR, L., and SOCHEN, N. Solving the functional eigen-problem using neural networks. arXiv Preprint, arXiv: 2007.10205(2020) https://doi.org/10.48550/arXiv.2007.10205
[24] JIN, H., MATTHEAKIS, M., and PROTOPAPAS, P. Physics-informed neural networks for quantum eigenvalue problems. 2022 International Joint Conference on Neural Networks IJCNN, Italy, 1-8(2022)
[25] RAISSI, M., PERDIKARIS, P., and KARNIADAKIS, G. E. Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, 686-707(2019)
[26] TANCIK, M., SRINIVASAN, P., MILDENHALL, B., FRIDOVICH-KEIL, S., RAGHAVAN, N., SINGHAL, U., RAMAMOORTHI, R., BARRON, J., and NG, R. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33, 7537-7547(2020)
[27] EDELMAN, A., ARIAS, T. A., and SMITH, S. T. The geometry of algorithms with orthogonality constraints. SIAM Journal on Matrix Analysis and Applications, 20(2), 303-353(1998)
[28] SALIMANS, T. and KINGMA, D. P. Weight normalization: a simple reparameterization to accelerate training of deep neural networks. Advances in Neural Information Processing Systems, 2016, 901-909(2016)
[29] JACOT, A., GABRIEL, F., and HONGLER, C. Neural tangent kernel: convergence and generalization in neural networks. Advances in Neural Information Processing Systems, 2018, 8580-8589(2018)
[30] WANG, S. F., TENG, Y. Y., and PERDIKARIS, P. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing, 43(5), 3055-3081(2021)
[31] KINGMA, D. P. and BA, J. Adam: a method for stochastic optimization. arXiv Preprint, arXiv: 1412.6980(2014) https://doi.org/10.48550/arXiv.1412.6980
[32] EVANS, L. C. Partial Differential Equations, American Mathematical Society, Washington D. C. (2010)
[33] JAFARZADEH, S., LARIOS, A., and BOBARU, F. Efficient solutions for nonlocal diffusion problems via boundary-adapted spectral methods. Journal of Peridynamics and Nonlocal Modeling, 2, 85-110(2020)
[34] LOPEZ, L. and PELLEGRINO, S. F. Computation of eigenvalues for nonlocal models by spectral methods. Journal of Peridynamics and Nonlocal Modeling, 1, 1-22(2021)
[35] ALALI, B. and ALBIN, N. Fourier spectral methods for nonlocal models. Journal of Peridynamics and Nonlocal Modeling, 2, 317-335(2020)
[36] WISKOTT, L. and SEJNOWSKI, T. J. Slow feature analysis: unsupervised learning of invariances. Neural Computation, 14(4), 715-770(2002)
[37] SPREKELER, H. On the relation of slow feature analysis and Laplacian eigenmaps. Neural Computation, 23(12), 3287-3302(2011)
[38] BERKES, P. and WISKOTT, L. Slow feature analysis yields a rich repertoire of complex cell properties. Journal of Vision, 5(6), 9-9(2005)
[39] BRADBURY, J., FROSTIG, R., HAWKINS, P., JOHNSON, M. J., LEARY, C., MACLAURIN, D., NECULA, G., PASZKE, A., VANDERPLAS, J., WANDERMAN-MILNE, S., and ZHANG, Q. JAX: composable transformations of Python+NumPy programs (2018)
[40] HUNTER, J. D. Matplotlib: a 2D graphics environment. IEEE Annals of the History of Computing, 9(3), 90-95(2007)
[41] HARRIS, C.R., MILLMAN, K. J., VAN DER WALT, S. J., GOMMERS, R., VIRTANEN, P., COURNAPEAU, D., WIESER, E., TAYLOR, J., BERG, S., SMITH, N. J., KERN, R., PICUS, M., HOYER, S., VAN KERKWIJK, M. H., BRETT, M., HALDANE, A., RIO, J. F., WIEBE, M., PETERSON, P., GERARD-MARCHANT, P., SHEPPARD, K., REDDY, T., WECKESSER, W., ABBASI, H., GOHLKE, C., and OLIPHANT, T. E. Array programming with NumPy. nature, 585(7825), 357-362(2020)

A dive into spectral inference networks: improved algorithms for self-supervised learning of continuous spectral representations

A dive into spectral inference networks: improved algorithms for self-supervised learning of continuous spectral representations

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	J. N. FUHG, A. KARMARKAR, T. KADEETHUM, H. YOON, N. BOUKLAS. Deep convolutional Ritz method: parametric PDE surrogates without labeled data[J]. Applied Mathematics and Mechanics (English Edition), 2023, 44(7): 1151-1174.
[2]	Xuhui MENG. Variational inference in neural functional prior using normalizing flows: application to differential equation and operator learning problems[J]. Applied Mathematics and Mechanics (English Edition), 2023, 44(7): 1111-1124.
[3]	Yichuan HE, Zhicheng WANG, Hui XIANG, Xiaomo JIANG, Dawei TANG. An artificial viscosity augmented physics-informed neural network for incompressible flow[J]. Applied Mathematics and Mechanics (English Edition), 2023, 44(7): 1101-1110.
[4]	M. KIM, G. LIN. Peri-Net-Pro: the neural processes with quantified uncertainty for crack patterns[J]. Applied Mathematics and Mechanics (English Edition), 2023, 44(7): 1085-1100.
[5]	Zhiping MAO, Xuhui MENG. Physics-informed neural networks with residual/gradient-based adaptive sampling methods for solving partial differential equations with sharp solutions[J]. Applied Mathematics and Mechanics (English Edition), 2023, 44(7): 1069-1084.
[6]	W. WU, M. DANEKER, M. A. JOLLEY, K. T. TURNER, L. LU. Effective data sampling strategies and boundary condition constraints of physics-informed neural networks for identifying material properties in solid mechanics[J]. Applied Mathematics and Mechanics (English Edition), 2023, 44(7): 1039-1068.
[7]	Yueping WANG, Lizhong MU, Ying HE. Thermogram-based estimation of foot arterial blood flow using neural networks[J]. Applied Mathematics and Mechanics (English Edition), 2023, 44(2): 325-344.
[8]	Lincong CHEN, Zi YUAN, Jiamin QIAN, J. Q. SUN. Random vibration of hysteretic systems under Poisson white noise excitations[J]. Applied Mathematics and Mechanics (English Edition), 2023, 44(2): 207-220.
[9]	Xiaochen MAO, Xingyong LI, Weijie DING, Song WANG, Xiangyu ZHOU, Lei QIAO. Dynamics of a multiplex neural network with delayed couplings[J]. Applied Mathematics and Mechanics (English Edition), 2021, 42(3): 441-456.
[10]	Xuhui WANG, Nanjing HUANG. Finite-time consensus of multi-agent systems driven by hyperbolic partial differential equations via boundary control[J]. Applied Mathematics and Mechanics (English Edition), 2021, 42(12): 1799-1816.
[11]	Zeyu LIU, Yantao YANG, Qingdong CAI. Neural network as a function approximator and its application in solving differential equations[J]. Applied Mathematics and Mechanics (English Edition), 2019, 40(2): 237-248.
[12]	王如彬张志康 Chi K. Tse. Neurodynamics analysis of brain information transmission[J]. Applied Mathematics and Mechanics (English Edition), 2009, 30(11): 1415-1428.
[13]	颜向平;李万同. Global asymptotic stability for Hopfield-type neural networks with diffusion effects[J]. Applied Mathematics and Mechanics (English Edition), 2007, 28(3): 361-361 .
[14]	张继业. GLOBAL STABILITY ANALYSIS IN CELLULAR NEURAL NETWORKS WITH UNBOUNDED TIME DELAYS[J]. Applied Mathematics and Mechanics (English Edition), 2004, 25(6): 686-693.
[15]	马军海;陈予恕;辛宝贵. STUDY ON PREDICTION METHODS FOR DYNAMIC SYSTEMS OF NONLINEAR CHAOTIC TIME SERIES[J]. Applied Mathematics and Mechanics (English Edition), 2004, 25(6): 605-611.