Balance Test

[Helz]

Its been a while sense we have done this. I plan on doing it myself but more information > less information and the results may be different depending on who is in game. Basically I would like to get a rough idea of how often what role/team wins in relation to an amount of games. If you would like to help out as you play just make note of who won and post it here. Something like this:

"I played 23 games, Town won 13 of them, Mafia/triad won 7, SK won 1, Cult won 2."

This will give us an idea of exactly how balanced the factions are right now.

30 Games-

7 town wins
10 Triad/Mafia wins
2 Cult wins
2 MM wins
2 Nobody wins
1 SK wins
2 Arson wins

[SuperJack]

I will start my count.

[lingsarecute]

Town wins most of the games unless the invests get unlucky or the town is dump and doesn't lynch or they follow mafia votes. Mafia is easy to win because you have allies, roles such as sk, mm will usually lose unless the gf gets lynched/killed(veteran, jailor). Arson has a decent chance of winning unless - ignores invulnerability is OFF. Winning as a neutral evil is hard because mafia will accidently kill you in most of the games. Neutral benign win - jester, executioner and survivor is easy. Amnesiac is usually hard - you will most likely die n1 or pick the wrong faction. Cult(I'm talking about cultist not witch doctor) can easily win the game if the cult doesn't get revealed - they will be revealed the last night in massive numbers. Also afkers and newer players can affect the game a lot.

[SuperJack]

Town wins most of the games unless the invests get unlucky or the town is dump and doesn't lynch or they follow mafia votes. Mafia is easy to win because you have allies, roles such as sk, mm will usually lose unless the gf gets lynched/killed(veteran, jailor). Arson has a decent chance of winning unless - ignores invulnerability is OFF. Winning as a neutral evil is hard because mafia will accidently kill you in most of the games. Neutral benign win - jester, executioner and survivor is easy. Amnesiac is usually hard - you will most likely die n1 or pick the wrong faction. Cult(I'm talking about cultist not witch doctor) can easily win the game if the cult doesn't get revealed - they will be revealed the last night in massive numbers. Also afkers and newer players can affect the game a lot.

This was completely unhelpful.

Its been a while sense we have done this. I plan on doing it myself but more information > less information and the results may be different depending on who is in game. Basically I would like to get a rough idea of how often what role/team wins in relation to an amount of games. If you would like to help out as you play just make note of who won and post it here. Something like this:

"I played 23 games, Town won 13 of them, Mafia/triad won 7, SK won 1, Cult won 2."

This will give us an idea of exactly how balanced the factions are right now.

Well, it doesn't show exactly how balanced the game is, because of the setups. Rather do it per role also . If it was targeted specifically e.g. Mass Murder won 3/10 games, SK won 4/13 games.

[yzb25]

The statistics you'll get from this simply won't be worth using. Firstly, hone in on 933 saves (or close deviants like 932 or w/e).

The glory of statistics is that they are good for finding the probability of specific, individual occurrences that are random or incalculable. A handful of statistics shouldn't be used to answer such broad, abstract questions (like the balance of a game) that have a mixture of cause/effect AND randomness (and questions that feature partial subjectivity). You need to be more precise with your queries (i.e. leaver statistics). Furthermore, you need to put rules in place that disregard anomalous games (like veteran killing half of the players on n1) without encouraging cherry picking.

Once you have deduced certain key probabilities through solid analysis, like town lynching mentality, who tends to visit who, and the like, you can then start to estimate the probabilities of different outcomes from a 933 game, perhaps using a few solid case studies for reference material.

If you don't go to this length, the statistics you find for who wins and who loses will be a meaningless waste of time, suitable only for lobbing at others in petty arguments. In my opinion, finding statistics on the probability that someone leaves, and what role the leaver tends to be is a very integral part of balancing pub saves that is rarely considered. This would be a far more objective and quantifiable pursuit THAT CAN ACTUALLY YIELD USEFUL RESULTS, rather than "herpderp who wins the most".

[Helz]

The statistics you'll get from this simply won't be worth using. Firstly, hone in on 933 saves (or close deviants like 932 or w/e).

The glory of statistics is that they are good for finding the probability of specific, individual occurrences that are random or incalculable. A handful of statistics shouldn't be used to answer such broad, abstract questions (like the balance of a game) that have a mixture of cause/effect AND randomness (and questions that feature partial subjectivity). You need to be more precise with your queries (i.e. leaver statistics). Furthermore, you need to put rules in place that disregard anomalous games (like veteran killing half of the players on n1) without encouraging cherry picking.

Once you have deduced certain key probabilities through solid analysis, like town lynching mentality, who tends to visit who, and the like, you can then start to estimate the probabilities of different outcomes from a 933 game, perhaps using a few solid case studies for reference material.

If you don't go to this length, the statistics you find for who wins and who loses will be a meaningless waste of time, suitable only for lobbing at others in petty arguments. In my opinion, finding statistics on the probability that someone leaves, and what role the leaver tends to be is a very integral part of balancing pub saves that is rarely considered. This would be a far more objective and quantifiable pursuit THAT CAN ACTUALLY YIELD USEFUL RESULTS, rather than "herpderp who wins the most".

We should agree to disagree then. I want to include games with leavers and afk players and all the rest of it because that is the norm in the mod and the norm of the mod is what we balance for. We have added countless roles that "Should" have been OP as all hell but were not because of how they are played. You can not break this game down to an exact science under perfect conditions for the purpose of balancing it for the community. You can however see how the game shakes out over 100 games or so.

This is just to get an idea if something is really out of whack so we will know that we have something to look into.

[Slaol]

Leavers reflect he realistic game one joins. Its unfortunate, but they are a factor and a reality. Thats just that. Helz is correct.

[yzb25]

You appear to have misunderstood me. I wasn't saying we should disregard leavers, I was saying that, to give these statistics any value, we'd have to isolate everything, consider it, and THEN bring it together.

If I was trying to balance a save with 7 sheriffs, 3 executioners, and 4 mafiosos and 1 framer, I'd calculate probabilties that framer successfully frames someone a sheriff checks, and how often. I'd also gather statistics for how often executioner successfully misleads on average and calculate how much of a detriment it is when an executioner successfully misleads on average, and so on. Once I understand the impact of all these variables individually, THEN I'd bring them together to provide a comprehensive answer to the likelihood one faction wins over another. The advantage of deconstructing it like this is that I'd understand to what extent everything affects everything else. It'd mean I can say: "well, if I add another framer and remove a sheriff, it will decrease town success by x%, according to my model".

Clearly, calculating things to such precision is effectively impossible for us. However, the problem with your approach of just "what are the winrates in every game ever?" is that it gives you no indication as to HOW those figures look that way. Imbalance is a strange thing. Imbalances cancel eachother out, not everyone exploits imbalances, imbalances can be general, imbalances are very elusive. Plus, the fact you can customize your save until you get something alright means that balance will generally be reasonable, even if it's unstable or lacks flexibility.

Taking this approach will tell you barely anything of value about the imbalance of the game - it will probably seem like there are no imbalances at all. It may give you an indication if something extreme is happening, but you'd be able to tell if something extreme was happening without these statistics, anyway. It'd simply be obvious on an anecdotal level.

That's why I proposed: take something everyone seems to have a very different experience with ("only 1 or 2 people leave my game ever and that's uncommon" vs. "omg I feel like 3 people leave per game") and give us a more objective perspective. That would be far more useful than some bogus statistic that won't tell us anything worth noting.

I don't mean to sound aggressive, but doing this truly is a waste of time and will tell you nothing of any value whatsoever. Your own anecdotes would have greater credibility than this statistic. Sorry for the wall of text :/

[SuperJack]

You appear to have misunderstood me, I wasn't saying we should disregard leavers, I was saying that, to give these statistics any value, we'd have to isolate everything, consider it, and THEN bring it together.

If I was trying to balance a save with 7 sheriffs, 3 executioners, and 4 mafiosos and 1 framer, I'd calculate probabilties that framer successfully frames someone a sheriff checks, and how often. I'd also gather statistics for how often executioner successfully misleads on average and calculate how much of a detriment it is when an executioner successfully misleads on average, and so on. Once I understand the impact of all these variables individually, THEN I'd bring them together to provide a comprehensive answer to the likelihood one faction wins over another. The advantage of deconstructing it like this is that I'd understand to what extent everything affects everything else. It'd mean I can say: "well, if I add another framer and remove a sheriff, it will decrease town success by x%, according to my model".

Clearly, calculating things to such precision is effectively impossible for us. However, the problem with your approach of just "what are the winrates in every game ever?" is that it gives you no indication as to HOW those figures look that way. Imbalance is a strange thing. Imbalances cancel eachother out, not everyone exploits imbalances, imbalances can be general, imbalances are very elusive. Plus, the fact you can customize your save until you get something alright means that balance will generally be reasonable, even if it's unstable or lacks flexibility.

Taking this approach will tell you barely anything of value about the imbalance of the game - it will probably seem like there are no imbalances at all. It may give you an indication if something extreme is happening, but you'd be able to tell if something extreme was happening without these statistics, anyway. It'd simply be obvious on an anecdotal level.

That's why I proposed: take something everyone seems to have a very different experience with ("only 1 or 2 people leave my game ever and that's uncommon" vs. "omg I feel like 3 people leave per game") and give us a more objective perspective. That would be far more useful than some bogus statistic that won't tell us anything worth noting.

I don't mean to sound aggressive, but doing this truly is a waste of time and will tell you nothing of any value whatsoever. Your own anecdotes would have greater credibility than this statistic. Sorry for the wall of text :/

Its what I wanted to say, but If I had said it, It would mean vast amounts of work.

But I totally agree with what you are saying, if we want to understand the Balance of the game, we need to do what you said.

Would you like to create the tests we must do to find balance?

[yzb25]

Its what I wanted to say, but If I had said it, It would mean vast amounts of work.

But I totally agree with what you are saying, if we want to understand the Balance of the game, we need to do what you said.

Would you like to create the tests we must do to find balance?

nah, my point was that this complex method, though the right way, is extremely difficult for us. So it's much better, if we want to do any stats at all, to just find more humble things and answer little questions here and there that help us. There's no rush to become the god of balance by tomorrow - let's just do it one baby step at a time. We should start with leaving, or something of similar magnitude.

EDIT: But if I did have support, this could be an intriguing summer holiday project.

[Slaol]

Well, i tend to believe my anecdotes are the end all be all, but i can see where others might disagree.

[Cryptonic]

Just have the stats save in bank and create a mod command that shows stats of every player in game for you. Lol

[Helz]

Well this is kind.... here is what I have so far:

30 Games-

7 town wins
10 Triad/Mafia wins
2 Cult wins
2 MM wins
2 Nobody wins
1 SK wins
2 Arson wins

None of the saves were troll and only 1 of them was an 11 player game. The rest were pretty normal.

[SuperJack]

nah, my point was that this complex method, though the right way, is extremely difficult for us. So it's much better, if we want to do any stats at all, to just find more humble things and answer little questions here and there that help us. There's no rush to become the god of balance by tomorrow - let's just do it one baby step at a time. We should start with leaving, or something of similar magnitude.

EDIT: But if I did have support, this could be an intriguing summer holiday project.

What sort of support would you like?

[Cryptonic]

You can't really change anything to the game to increase balance, anyways. It's all in the setup, which has been nearly "perfected" over the years. Players just need to balance KPN to the number of towns so they're not annihilated lol. Also, games that end up having more than one town killing are pretty much a joke, as well (excluding BG, ect). Killing a Non-Town is a complete game changer, but a Town killing another Town also is; adding multiple Town Killing hurts balance.
People have pretty much got acustomed to 9-3-3 as the magic ratio for Mafia lol. I don't agree with this, but no one is willing to use Citizens so it's as close as we can get :P

[creedkingsx]

Any ideal game should result in a Town win. Given a larger pool of town players, mistakes are bound to happen and it is the job of all non-town factions to capitalize on these mistakes.
Setups should be in favor of town winning, obviously not so much that it is an auto-win, but enough to create the potential for mistakes as a result from the number of them.

[MasterNinja]

town wins 60-70% of all games. and thats that.

[Cryptonic]

town wins 60-70% of all games. and thats that.

Well, there you go Helz
/close

[MasterNinja]

i was being serious. if you have around 1000 games, the town will win Atleast Half of them!

[Cryptonic]

i was being serious. if you have around 1000 games, the town will win Atleast Half of them!

And that's really great and accurate science, which is why we can now close this topic.

[MasterNinja]

hey dont take me so lightly.
if you have 5 games then town in average would win 3 of them.

i wonder what you would do to actually balance it. currently the game is making you a good laugh with funny names and nice talk about anything.
people becoming murder roles that is immune (that was my favorite in the beginning of the game cause then i knew i would not die early days and could kill whoever i wanted)
but half a year ago or so i started to hate being killer roles etc and just blacklisted most of it.
because its boring, you cant (you can) talk up to town and give information about who is killer roles or non towns and make town follow you. and if you get lynched the town wont feel bad XD.

my new role for town is King, i figure out players roles by reading the chat and people's attitudes shown in the chat. and when i lynch a mafia / neutral evil in for example day 2 or day 3 when i am doctor or survivor. (love being survivor) then people follow me and maf rage.
even tho saying clues on day 2 somethimes you can get urself lynched a couple times... =)

so im kinda confused with the topic. (in the matter of why). the game is great fun, it doesn't feel unbalanced. its alot up to the players you come with and the save people gives. its a good joy playing mafia that the other games doesn't have and its more of all the feels and chatting.
and one part of the internets greatest power is to be unknown. no one knows who you are but you are just there. and you can do whatever you want. and everyone in mafia has different attitudes and different way of speach and we talk about all sorts of topics somethimes that isn't even about the game. its Le Amazing

[Cryptonic]

hey dont take me so lightly.
if you have 5 games then town in average would win 3 of them.

i wonder what you would do to actually balance it. currently the game is making you a good laugh with funny names and nice talk about anything.
people becoming murder roles that is immune (that was my favorite in the beginning of the game cause then i knew i would not die early days and could kill whoever i wanted)
but half a year ago or so i started to hate being killer roles etc and just blacklisted most of it.
because its boring, you cant (you can) talk up to town and give information about who is killer roles or non towns and make town follow you. and if you get lynched the town wont feel bad XD.

my new role for town is King, i figure out players roles by reading the chat and people's attitudes shown in the chat. and when i lynch a mafia / neutral evil in for example day 2 or day 3 when i am doctor or survivor. (love being survivor) then people follow me and maf rage.
even tho saying clues on day 2 somethimes you can get urself lynched a couple times... =)

so im kinda confused with the topic. (in the matter of why). the game is great fun, it doesn't feel unbalanced. its alot up to the players you come with and the save people gives. its a good joy playing mafia that the other games doesn't have and its more of all the feels and chatting.
and one part of the internets greatest power is to be unknown. no one knows who you are but you are just there. and you can do whatever you want. and everyone in mafia has different attitudes and different way of speach and we talk about all sorts of topics somethimes that isn't even about the game. its Le Amazing

Spoken like a true non-FM player.

[creedkingsx]

Spoken like a true non-FM player.

Unfortunately, it's not fair to group FM and SC2 at any capacity. They are two entirely different skill levels.

[MasterNinja]

so u dont agreed that i think there should be no balance changes done and the game is pretty much finished?
cause thats what it sounds like.

stick to topic son.

[Slaol]

Unfortunately, it's not fair to group FM and SC2 at any capacity. They are two entirely different skill levels.

One is Bronze, where the masses and masses of terrible games give nice overall opinions on the game.
One is Chellenger, where even the worst players steamroll the lower league and we pretend we are a well oiled machine.

[yzb25]

What sort of support would you like?

The not-so-fun kind. The 'supporters' would likely be given a questionnaire with precise, rigorous questions and they'd have to fill this out for every game they watch, with exceptions noted down. I'd want a solid 150 (all 9-3-3 to keep things simple) games' worth of data, too. It sounds daunting, but if 10 reasonably reliable people could be found it would only take one boring day or a couple of boring evenings. Getting this many games is within reach, too.

The questionnaire would focus on the parts of the game that can't be calculated to a reasonable degree with pure maths (lynching mentality, who tends to visit who, leaver statistics, exc.) and thus probabilities are needed instead.

As has already been stated, the natural "perfecting of the setup" means that the balance between the alignments is tolerable, but understanding what truly causes the alignments to all do okay (whether it's players carrying teams, people from many alignments all dying by fluke, exc.) would be very useful for future changes and understanding the effect new roles could potentially have on the meta and saves.

Again, however, it'd be a summer project. I'm talking 3 or 4 month's time.