Grimrock

The Short Version:

If you're sticking with Light armor and want to know whether the Lurker set or Chitin set is best in terms of withstanding attacks: it's the Lurker set. If you're willing to put some points into Heavy armor, Plate or Valor will perform better than Lurker. Point-for-point, Evasion seems to be better than Protection. Also, I guess I can get obsessive about stuff?

The Long Version:

I had been going through a Toorum run of Grimrock (just on Normal difficulty), intending to just stay with Light Armor for him, and I had been undecided whether to go with the Lurker set or the Chitin set. I had seen a few threads on here (and elsewhere) claiming that the Lurker set was the way to go, but I hadn't actually seen any numbers. (I'd also come across at least one instance of someone saying that "evasion wasn't cutting it" or somesuch.) I had been keeping items from both sets, as I found them, but when I found the third Lurker item via Level 6 (and with another Chitin to find later in the level), I figured that perhaps it'd be informative to collect some data and figure it out definitively.

At the time I was right around the Uggardian-infested southwest pit area on Level 6, and hadn't yet dropped down to fight the crab guarding the Crown of Kings - I figured that'd be as good a place as any to test. Toorum was Level 9 at the time, with 165 health and a Dex of 16. His base Protection was 1 (thanks to the first Armors skill) and his base Evasion was 10 (thanks to Dex). I cheated a bit to get myself the full sets of armor, unequipped everything else (so the armor set was the only factor being changed), dropped down into the pit, chugged down a health potion to get back to the full 165 health, and then saved the game before the gate had a chance to open. I then basically just did a number of runs keeping track of how many times the crab attacked versus how many times I got hit until I was dead. The crab actually makes two attacks with each lunge - fortunately it's easy to tell which one killed you by the frozen animation frame (gaping crab-maw == first attack, claw covering up its "face" == second attack). I did that twenty-five times for each, figuring that should hopefully even out most potential PRNG quirkiness.

The data ended up looking like this:
Chitin Set (Chitin Mail, Chitin Greaves, Chitin Boots, Chitin Mask) (Protection: 37, Evasion: 10)
Avg Hit Ratio: 62% (std dev: 14.7)
Avg Hits: 6.1 (std dev: 0.7)
Avg Attacks: 10.1 (std dev: 2.5)
Avg DPH: 27.1 (std dev: 3.4)

Lurker Set (Lurker Pants, Lurker Vest, Lurker Hood, Lurker Boots) (Protection: 1, Evasion: 30)
Avg Hit Ratio: 42% (std dev: 12.6)
Avg Hits: 4.7 (std dev: 0.4)
Avg Attacks: 11.6 (std dev: 2.8)
Avg DPH: 35.2 (std dev: 3.6)

Those are just arithmetic means - I took a look at geometric but the numbers were nearly identical and didn't change the comparative results at all. "DPH" is "Damage Per Hit," which is bound to be a bit off because that number sort of assumes that the crab dealt exactly 165 points of damage, when in reality it would have inflicted more than that, but it's good enough for comparative purposes.

I figure that the "Average Attacks" is really the most important number of the bunch, and the Lurker set seems to, on average, let you survive a little more than one attack more than the Chitin set would do. The standard deviations seem reasonably constant as well (and some previous, less rigorous testing seemed to bear that out as well), so I think it's pretty good.

Once I had found that, I actually got to wondering how the heavy armors would compare as well, especially in a corner case where perhaps someone didn't want to actually put the skill points into Heavy Armor, thinking that perhaps the added Protection from a full Valor set might offset the strong Evasion hit. So, after more testing, this is what I got:

Plate Set, no Heavy Armor expertise (Plate Cuirass, Plate Greaves, Plate Gauntlets, Plate Boots, Full Helmet) (Protection: 61, Evasion: -40)
Avg Hit Ratio: 87% (std dev: 8.9)
Avg Hits: 7.9 (std dev: 1.6)
Avg Attacks: 9.0 (std dev: 1.9)
Avg DPH: 21.4 (std dev: 4.5)

Valor Set, no Heavy Armor expertise (Cuirass of Valor, Greaves of Valor, Helmet of Valor, Gauntlets of Valor, Boots of Valor) (Protection: 76, Evasion: -40)
Avg Hit Ratio: 83% (std dev: 11.4)
Avg Hits: 9.4 (std dev: 2.3)
Avg Attacks: 11.5 (std dev: 3.7)
Avg DPH: 18.5 (std dev: 5.3)

Plate Set, with Heavy Armor expertise (Plate Cuirass, Plate Greaves, Plate Gauntlets, Plate Boots, Full Helmet) (Protection: 61, Evasion: 10)
(I used the console to bump Toorum's health back down to 165, since the step inbetween Light and Heavy in the skill tree provides +10 Health)
Avg Hit Ratio: 64% (std dev: 10.4)
Avg Hits: 8.6 (std dev: 1.3)
Avg Attacks: 13.9 (std dev: 3.8)
Avg DPH: 19.1 (std dev: 2.8)

So, without Heavy expertise, the Valor set performs just about as well as a Lurker set does (in terms of number-of-attacks-survived), whereas with Heavy expertise, the Plate armor does very noticeably better than Lurker. The moral of the story being that you shouldn't be using Heavy armor unless your character can support it. :) I didn't bother to do a run with Heavy-Armor-expertise Valor, since the Plate armor did so well and I was already a bit bored with the exercise. I'm guessing that the reported Valor hit percentage is actually a bit low... I suspect that some of the crab's attacks technically hit but did zero damage thanks to the Protection. I certainly did see more than one 1-damage hit while wearing Valor armor.

Anyway, there you have it. I've put up a CSV of the raw data in case anyone felt compelled to check my math, etc: http://apocalyptech.com/grimrock/armor-data.csv

Enjoy!

Wow, that's quite an in-depth analysis! I think the general consensus is that we sort of already knew this, but seeing the numbers crunched really brings it home, nice

Nice! There's a little flaw, though. You compare a 4-pieces set with a 5-pieces set... but a normal player won't just go around without wearing gloves with his lurker set. At least you've taken the shield off the valor set :p.
Nice work overall anyway.

FlashSoul wrote:Nice! There's a little flaw, though. You compare a 4-pieces set with a 5-pieces set... but a normal player won't just go around without wearing gloves with his lurker set. At least you've taken the shield off the valor set :p.
Nice work overall anyway.

Ah, yeah, I had meant to mention that as well but forgot amidst everything else. I realized that about the gloves when I was most of the way through the last heavy-armor test, and just didn't care enough to re-do 75 more runs. :) I think in general the comparison works out anyway, though - the only real change would be pushing Lurker a bit closer to the Plate-with-Heavy-expertise numbers. The best you'd be able to do with light armor for gloves is, I think, the Pit Fighter Gauntlets, which'd give you a +3 Protection (which actually isn't even classified as Light). Perhaps I'll do a run with Lurker+Pit-Fighter-Gauntlets tonight...

Edit: Though actually, now that i think about it some more, that actually brings up some other interesting questions. Given my conclusion that a point of Evasion is worth more than a point of Protection, how exactly are they weighted? If I remove the Plate gloves (effectively penalizing me with -12 Protection but with a benefit of +10 Evasion), would that be a net gain? (Assuming we're playing around with the no-Heavy-expertise scenario, anyway.) So a Plate set without the gloves could be better than a Plate set WITH the gloves? If so, how would having literally NO armor compare, versus using Plate without Heavy expertise? A quick 5-run of no armor at all yields:

Avg Hit Ratio: 58% (std dev: 17.0)
Avg Hits: 4.6 (std dev: 0.5)
Avg Attacks: 8.4 (std dev: 2.2)
Avg DPH: 36.2 (std dev: 3.9)

... though I'd like to see that on a larger sample size. Looks like I may have talked myself into running some more of these tonight. :)

I suspect that Grimrock's math might work a little differently when they're in the negative values - if I take the Lurker and Chitin numbers, based on the Average Attacks (37p+10e=10.1 and p+30e=11.6), p=.17 and e=.381 - so Evasion is double what Protection provides. That actually follows Plate-Plus-Heavy-Experience reasonably well (61p + 10e = 14.1), but when the Evasion is negative, that gets thrown off (also it doesn't really fit that no-armor run). It's entirely possible too that I'm going about that all wrong. I haven't really put much thought into it... I suppose it's quite probable that the benefits/costs aren't linear, to boot.

Edit Edit: And, I suppose, it's entirely possible that some of this depends on which monster you're fighting, too. Perhaps someone at Almost Human will drop us the sourcecode for monster attack resolution and we can just programmatically figure out what's best in what circumstances. :)

Nice job!

Two things that may influence the evaluation:

1. Many monsters have an 'accuracy' stat (or better said, their accuracy is above normal). I assume evasion will be less efficient when fighting monsters with increased accuracy, while protection always has the same effect. Additionally there are monsters with ranged attacks, and ranged attacks never miss.

2. Evasion is completely based on luck. It's either all or nothing. While it doesn't matter in the long run (as it will even out statistically) it does matter during a single fight. If you're dead, you're dead and you will have to reload. So even with a high evasion value you will get hit often if you're out of luck and eventually die, while protection always reduces the damage done. The amount of damage reduction may also be luck based, but it is most likely a percentage value and thus it will never be zero.

That's why protection is more reliable imo, while evasion is statistically better, but unreliable.