Well, partly true...they were not clones of Fett, and as you say neither were the vast majority of stormtroopers after Episode 3. And the Imperial Guards did not exist prior to the empire, however, the Senate's Guards (And Supreme Chancellor's guard) did, and the latter were who they adopted the Uniform off, though arguably they were only a renaming of the Red Guard anyway.

Depending on what you refer to, Stormtroopers later weren't clones for one of a couple reasons, either its to do with an insurrection on Kamino, or the sheer lack of needing them to be a cloned force anymore when you have the population of the majority of the galaxy to draw recruits from.

Anyway, there is only a single piece of evidence to say they were clones, and quite a lot to say they weren't. And the fact they have been described as being "Hand picked" and "some are latent force sensitives" generally implies any clones amongst them were the exception rather than the rule.

there is also the reason that the kaminoins themselves after the formation of the empire they started to clone another army to defend kamino against Imperial control

non est innocentia, solum sunt noxiae gradus, There Is No Innocence, Only Degrees Of Guilt