A numerical scheme is developed to extend the scope of the spectral method without solving the covariant and contravariant forms of the Navier-Stokes equations in the curvilinear coordinates. The primitive variables are represented by the Fourier series and the Chebyshev polynomials in the computational space. The time advancement is accomplished by a high-order time-splitting method, and a corresponding high-order pressure condition at the wall is introduced to reduce the splitting error. Compared with the previous pseudo-spectral scheme, in which the Navier-Stokes equations are solved in the covariant and contravariant forms, the present scheme reduces the computational cost and, at the same time, keeps the spectral accuracy. The scheme is tested in the simulations of the turbulent flow in a channel with a static streamwise wavy wall and the turbulent flow over a flexible wall undergoing the streamwise traveling wave motion. The turbulent flow over an oscillating dimple is studied with the present numerical scheme, and the periodic generation of the vortical structures is analyzed.