The code you listed will return the number of alive enemies. You can use it directly in a damage formula. For instance, if all you had in the damage formula was: $gameTroop.aliveMembers().length - then you'd be doing damage equal to the number of enemies (before defense, etc..)

For actors you'd use $gameParty.aliveMembers().length

Thanks for confirming and providing the allied version. Glad to hear there's no magic math behind the scenes.

For anyone interested, I managed to figure out how to achieve the effect I wanted in the original post:

(<damage formula>) - (((<damage formula>) * ($gameTroop.aliveMembers().length - 1)) * 0.1)

The "0.1" (10%) represents the penalty. At 8 enemies, there is a 70% reduction to the ability's effectiveness (if it's an AoE, the total net damage would still be higher than hitting a single enemy, however).

If you want the first enemy to be taken into account also for the penalty (for whatever reason), you would simply use:

(<damage formula>) - (((<damage formula>) * ($gameTroop.aliveMembers().length)) * 0.1)

**If you calculate your defense formulas inside the skill formula box, you may need to put it at the end of the above formulas.*
And lastly, if you want to use this for healing effects, it would be this:

(<healing formula>) + (((<healing formula>) * ($gameParty.aliveMembers().length - 1)) * 0.1)

This would increase the healing by 10% per alive ally that is (technically) not the user.

Hopefully the formulas gives someone an idea or two on some new abilities.

Anywho, this is solved!