I'll brake the formula for you =)

A fictional example:

I have 1000 hp and 100 armor.

My opponent has 80 attack damage, 20 armor penetration, and 10% armor penetration.

1. Flat pen is applied first. Your armor goes down by 20. You have 100-20=80 armor left.

2. % pen is applied second. Your armor goes down by 10%80=8. You have 80-8 = 72 armor left.

72 armor increases your effective hp by 72%. This means that your oponent would need to do 1720 damage to kill you. You can find the % from here. 1720* X% = 1000

72 armor provides 42% damage reduction.

He hits you for 80, you take 80*58% = 46,5 damage.

Works exactly the same way for magic resistence/penetration.

If the armor is negative, it is calculated by the same formula, but instead it amplifies the damage (in place of reducing it).