The customer is (not) always right
“Your feedback is very valuable to us!” — Random Comunity Manager
I still remember the launch of the first big feature that I designed. I haven’t slept much the night before. I was tossing and turning in my bed worrying about all the possible and impossible ways that things could go wrong. Most of all I was worried about players’ reactions. Will they understand my design? Will they find it fun? Will they play it?
Six and a half years later, and I still don’t sleep before each update. For a game designer, the launch day is a bit like climbing on a rock stage, except that our audience can routinely be hundreds of thousands or even millions of people. It is our creation that we place in front of all these people. Every single one of these people will form an opinion about our work, at least, to a degree, whether they like it enough to keep playing it or not. Others, a minority to be sure, but a vocal one, will form an opinion strong enough that they will feel the urge to express it.
For a game designer, opening a Reddit page or checking Twitter for feedback about his game can be a horror-inducing experience. Internet is full of testimonies of game designers that got traumatized by player feedback. The audience can be harsh. People seem to be more willing to express negative emotions than positive ones. Maybe it is the zeitgeist, maybe it is our society, maybe it is basic human nature. I am not going to lament over it.
Running a game as a service for six and a half years helps. Running a free-to-play game helps. It helps in many ways, but most importantly it helps because gather a lot of data, enough of it to get to know our audience pretty well.
There are essentially two ways in which we gather player feedback:
- through telemetry data about player behavior
- via messages that players write about our game on social media, our own online forum, or send directly to our customer support.
Your opinion is very valuable to us, is not just a corny phrase. We do actually deeply care about player feedback. We care about the quality of our own work and have a desire to do better, but we also care about the happiness of our players. Happy players stay longer with our service and spend more money. Player retention is one of our key performance indicators.
Types of feedback
Those two ways in which we gather player feedback produce two fundamentally different types of feedback:
- Quantitative feedback
- Qualitative feedback
Quantitative feedback data is everything that can be measured directly by observing how players interact with the game. This includes things like engagement or the percentage of players that interact with a particular game feature, session length, retention, or the percentage of players that continue to engage with the feature after a specific number of days, etc. It can also include, things that are more specific to a particular feature. For example, the level that an average player has reached, the average score, the average amount of some virtual resource, etc.
Quantitative data is great. Numbers do not lie. They are objective measures of something. However, they do not tell the whole picture. They might tell you that something is happening within your game, but they will not necessarily tell you why is it happening.
Interpreting the data
To get the full picture one needs to take into account the other type of feedback, qualitative feedback, i.e. the stuff that players choose to tell about the game and their own experience playing it. As I mentioned, there is a multitude of ways players can express their opinion about the game, there are star ratings in app stores, official forums, customer support messages, official Facebook pages, and unofficial Facebook groups, Reddit pages, and Discord servers, Youtube videos, and comments, of course, Twitter.
It might so happen that you release a new feature or an update for your game and that both of these two types of data go hand in hand. Players engage heavily with your feature, they spend time and money on it, and they are very happy about it, expressing their joy and gratitude on social media. Presumably, your customer support will not have much work in this scenario.
Well, this is all fine in theory, but in practice, things do not always go this way. You might also think that this is the only good scenario. However, this is also the only scenario in which you do not learn anything. All your basic assumptions about your design have been proven true, players are happy and there is not much you should improve. Tinkering with the design further would probably cause damage.
But what do you do if things do not go this way? Of course, one possibility is that things will go horribly wrong and that both types of feedback would still be painting the same picture. The engagement is low and dropping, retention is terrible, players are raging on social media, and your customer support is flooded with complaint tickets. This might sound like a horrible turn of events. To be sure pissing off your player base is never a good thing. However, the silver lining here is that the combination of quantitative and qualitative data will most likely help you pinpoint and diagnose the problem. Unless you are Ed Wood of gaming you should be able to form an action plan and fix things somehow.
When feedback doesn’t align
The third scenario is the most difficult and the most interesting one. It is everything in between. It is a scenario in which two types of feedback paint a very different picture. It has several subtypes that can have a very different effect on the game.
It often happens that your players seem really happy with the content or the feature you have just released. They will write glorious reviews on Reddit and share screenshots on Facebook etc. etc. Yet the qualitative data will tell you that your feature is just not performing, the engagement might be low, or retention might be dropping, or monetization will be lacking. This might seem like a paradox. If they are happy why isn’t this obvious from data?
The answer to this is simple, only a tiny minority of people will care enough about your work to voice their opinions. All qualitative data that you can get originates with this vocal minority. The silent majority of players, that generate the bulk of qualitative data do not share the same impression. Fixing this situation can be extremely difficult. In order to lift your key performance indicator numbers, you might be required to alter exactly the things that the vocal minority likes. They might react negatively only to reinforce the already negative feelings of the silent majority, pushing you down the spiral of doom.
KEY IDEA: Qualitative feedback originates with the vocal minority, qualitative data originates with the silent majority.
There is of course a vice versa scenario. Your players are crying bloody murder yet all your KPIs are going through the roof. This obviously seems like another paradox. If they hate it why do they keep playing it? Why aren’t our quantitative indicators tanking? Should we do something about it? Should we panic?
The key to understanding this situation is human nature. What we are dealing with here are actually three quite distinct things:
- Things that people are willing to say aloud.
- Things that people are willing to admit to themselves
- Things that people are actually thinking, feeling, or doing
We humans are not particularly good at articulating our own emotions. Quite the contrary we are quite good at making the mess of them.
KEY IDEA: When facing something new, some people will express negative emotions even if they are actually impressed.
Some people are just like my mother-in-law. For them expressing negative emotions mean that they are actually impressed by something. For others, negative emotions might mean that they see the change as a new gameplay challenge. It means that they see what you did, see a new challenge, and are stepping up to it. This is especially true if your new feature is forcing the veteran players to rethink their strategies and relearn the skills that they already acquired. This might sound scary but is actually a very good thing! Just like muscle ache after an intense exercise, this is a sign of new growth.
If you are facing this scenario, my advice is to sit down and relax. Let the dust of the launch day settle. Both the opinions of players and telemetry data will evolve in a matter of days. Most likely the two types of feedback will start to converge into something actionable. Making rushed decisions in this scenario can lead you into an even deeper mess.
KEY IDEA: People do one thing, say another thing to themselves and say aloud yet a completely different thing.
The last scenario that I want to mention is the one that I personally dread the most. It is a scenario of mediocre emotions and mediocre or non-existent impact on quantitative data. Imagine a situation in which you have been working hard on the content or a new feature for several months. The launch day finally arrives but all you get from your players is a big giant “meh”!? Your effort has essentially been wasted. There is nothing to fix cos there is nothing that needs to be fixed. Players kinda appreciate your effort but are not really falling head over heels for it. There is nothing broken in your system, it is just not whooming with enough oumph.
KEY IDEA: It is not so much about if emotions are positive or negative, it is about the strength of emotions. Amplitude, not the direction!
In my personal experience, it is not about the direction of the emotion, but about the amplitude. It doesn’t matter if players are angry or happy about something as long as they feel emotions strong enough to wish to express them loudly. If you get an intense emotional response and it shows in your telemetry data, it means that you are onto something!
Key takeaways
- There are two types of feedback data.
- Quantitative data is everything that you can measure directly including engagement, retention, session length, score, level, speed of progression, etc.
- Qualitative what people say about your game, either directly to your customer support or on social media.
- To get the full picture of players’ opinion of your design you need to combine both types of data.
- What people actually do, what people say aloud, and what people admit to themselves are three very different things.
- Some people will express negative emotions even when they are actually positively surprised.
- It is not so important if emotions are positive or negative as long as they are strong and that quantitative data is good.
- “Meh” is the worst response you can get from players.
Links
