Nesting. Like $f(g(h(x)))$ where $f, g, h$ are functions? This is called function composition. The study of the iterations of a given function $f$ is called iterated functions. This overlaps with the study of fractals. For example the famous Mandelbrot set arises from the successive iterations of a particular function in the complex plane.