I am looking at the numbers from a relative perspective because that is what matters if you want to maximize your DPS. Increasing your expected damage from 100 to 110 is a lot different than increasing it from 500 to 510.

DPS stats like crit chance, crit damage, AD, and ASPD all scale linearly by themselves, but stack multiplicatively with each other. The more of one stat you stack, the less effective it is to continue stacking that stat, and the more effective it is to stack other dps stats. If you already have 250% crit damage (from IE), it is not cost effective to continue stacking more of it via runes but it is better to get more of another stat such as crit chance and armor pen.

Armor pen is different, the more of it you stack, the more effective it is (I believe the growth is hyperbolic until you reach 0). The benefit you gain from it also stacks multiplicatively with the other dps stats. If someone has 30 armor (early game stats) and you have 30 armor pen (marks and quints only) the damage you deal is increased by 30%. This is a huge boost and the reason you see most people using armor pen marks on physical carries. This does taper down a bit late game when people have a chance to get armor, but even when the target has 130 armor (late game non-tanks) your 30 ArPen still translates into a 15% damage increase, still very significant for only using marks and quints.

Even on a guaranteed crit, armor pen runes will make you do more damage than you can get from crit damage runes unless the enemy is heavily stacking armor, but you shouldn’t be focusing the tank first anyways. Armor pen also has the benefit of affecting your non-crits and physical abilities as well. It also works against towers (which you can’t crit).

Critical strike is good for spike damage, which can catch an enemy off guard. If you want more spikes, get crit chance runes, if you want harder spikes and harder normal hits, get armor pen. If you build IE early, get crit chance. If you get brutalizer, youmuu’s, starks, and/or black cleaver get armor pen. In no case whatsoever should you ever get crit damage runes.

The math has been done on this many times, but it is common consensus that ArPen runes > crit chance > crit damage. There are some exceptions to this rule such as Trynd preferring crit chance to get bloodlust stacks as was mentioned by Issege. Attack speed runes can be good for jungling since the neutral creeps are nice enough to stand still while you attack them unlike most smart enemy players.

TL.DR. Get armor pen marks and quints