Does there exists a brach of mathematics, if exists, perhaps in computational mathematics, researches the properties of multiple nesting functions where every nested function is somewhat simple, such as piecewise linear function?

Does there exists a brach of mathematics, if exists, perhaps in computational mathematics, researches the properties of multiple nesting functions where every nested function is somewhat simple, such as piecewise linear function?

Nesting. Like $f(g(h(x)))$ where $f, g, h$ are functions? This is called function composition. The study of the iterations of a given function $f$ is called iterated functions. This overlaps with the study of fractals. For example the famous Mandelbrot set arises from the successive iterations of a particular function in the complex plane.