I'm asking HOW it is ambiguous? I don't know any other way to interpret $\cos(x)^3$. On the other hand, $\cos^3(x)$ IS ambiguous because the same notation is used for iteration. This makes complete sense since composition is the "multiplication" in the algebra of functions.

To further illustrate why the convention for trig functions is wrong, consider the notation $\cos^{-1}(x)$. This is usually interpreted to mean the functional inverse (again agreeing with the exponent here denoting function iteration). However, this violates the convention that an exponent here means exponentiation of the real number, $\cos (x)$. By definition this makes this notation ambiguous. This is typically resolved by just specifying what you mean in any context where it might be misunderstood.

However, I still don't see the ambiguity in writing $\cos(x)^3$. What other convention does this conflict with?

I think you have it backward. My high school days were a long time ago so I can't comment on contemporary trends in trig notation. But we were taught $\cos^3 x = (\cos x)^3$, and $\tan^{-1} x = \arctan x$, and that "Even though that's inconsistent, we all just get used to it." And of course at this level the concept of iterated function composition doesn't come up. so there's no ambiguity there.

YOU might be confused because of iterated function composition; but THEY could never be, because they probably haven't studied it.

So now we come to the expression in question, $\cos(x)^3$. When I see that, my first thought is:

If they meant $\cos^3(x)$ they'd have written that. So they must mean $\cos(x^3)$. In which case why did they write it as $\cos(x)^3$. Either way, whichever interpretation they intended, there was a clearer and more correct way to write it.

It doesn't matter what the "right" answer is. What matters is that reasonable people can put forth reasonable arguments either way. My position is based on my belief that the $\cos^3 x$ notation is so

*universally used at the level of trig class*, that if the teacher didn't write that then why not?

After all I don't know any rule of precedence that disambiguates $f(x)^3$. And if there isn't such a rule, then the expression is ambiguous. Either write $(f(x))^3$ or $f(x^3)$, depending on which you intend. Ambiguous notation is just careless; and especially so in in teaching.

The mere fact that this question generates any discussion at all; let alone genuine disagreement; shows that the teacher used bad notation. If you forced me to make a choice I'd say $f(x)^3$ means $(f(x))^3$. I'd agree with your interpretation. But I personally would never write notation like that. I'd be explicit one way or the other. And I would never expect a trig student to sort all this out. Clarity is always to be preferred, especially in teaching.

tl;dr: Ambiguous notation is always to be avoided, both in one's own work and especially in teaching. If a piece of notation generates a two page discussion here, it's ambiguous. Therefore teacher's wrong no matter what the expression means.