No, you don't take damage to them - the numbers don't change, and it's quite possible to have other boons that cancel or outweigh that. It's not damage because it doesn't meet the game's definition of damage, and doesn't match the common conception of "damage to attributes" where the numbers go down and come back up.
Further, death spiral? Hardly. 5 States a spiral doesn't make; each condition can only be taken once, and only two of them have direct combat relevance (STR & AGL), you have only three relevant states other than unimpaired by conditions that are relevant: str down, agl down, both down, and all filled +1 more taken, which is KO. There's also little lingering effect, no HP loss penalties.
SO not really a spiral, especially since only a subset do conditions.