By: Patrick W. Zimmerman

So, it’s no secret that Twitch has long had a sexism problem.  Women streamers have to put up with toxicity and (often) outright abuse everyday from the community populating the chats on their streams, according to periodical and academic inquiry as well as the ol’ eyeball (seriously, just take a look around the site’s own top streams page for some sexist language.  Go ahead, we’ll wait; it won’t take you very long.).

The real question, then, isn’t “is Twitch a happy utopia of equal treatment, where streamers hold hands in mutual respect and universal gaming access, sunshine, and/or rainbows?”  No, we’re going to lower our sights a bit to something along the lines of: Has it gotten any better on Twitch? Is increased awareness of the problem, both on the part of the platform itself and its users, changing behavior?

The Question

For part 1 of this project, we’re going to compare the frequency of certain sexist terminology in Twitch chats between a group of 50 female streamers to streams of their male counterparts.  What kinds of language are more frequently associated with audiences for female streamers?

Warning – This project will contain a lot of offensive and horribly hate-filled language as a necessary result of the subject of the study being the use of exactly that language in a particular online gaming community (Twitch.tv). 

Misogyny and swearing lie ahead. Ye be warned.

The short-short version

Women streamers still see a significantly higher perecntage of sexist and abusive language than their male counterparts. While not all of it is necessarily targeted at the streamer, it’s still much more present in chats. In other news, sky still blue, Pope still Catholic.

There are quite a few terms that show up way more often on women’s streams.

• Slurs reflecting stock female personality tropes: bitch or biatch, ditz or dits
• References to streamers’ butts
• cute or kawaii
• hair or other references to someone’s physical appearance such as look, hot, pretty, beautiful, fat, or thicc.
• Helpfully pointing out the fact that a woman streamer was, in fact, a girl.
• love you or love u

Exactly 3 qualifying terms showed up more often on male streamers’ chats than on female streamers’:

• Insulting someone’s intelligence (idiot, moron, retard, stoopid, or stupid).
• xxx or porn
• dick or cock (never saw that one coming).

Of the 23 qualifying terms, the most-frequent insult or sexist comment on male streams (butt comments, normalized frequency of 0.05) is used less often than the top 6 on female streams (about half as often as women streamers see the same term, 0.11).

The results

Here we go….

The important thing to pay attention to is the distance from the reference line / colors (on the graph) or the colored column (on the table). This is the differential in a term’s frequency between female and male streams. More pink / positive number

= more common among women-hosted streams.  More blue / negative number = more common among men-hosted streams.

Mouseover for details.

Or, for those of you less visually inclined, who are on mobile devices (where mouse-over info doesn’t really work great), or who just like to see all the details in one big-ass table (click to expand):

Notes on reading the table: Like all Principally Uncertain’s text-parsing projects, the terms are listed using Perl-flavored Regular Expressions, which allows our Ubuntu-hosted text processing scripts to parse simple logic like “x or y” (x|y), “x following a non-letter” [^a-z]x, etc. 

The fine print – For terms with a greater than 0.01% normalized frequency in either the male or female datasets, a Mann-Whitney U-test resulted in a z-value of 3.01714 (greater than the critical ±2.58 for 2-tailed tests with α=0.01) and a U of 156.5, showing significant gender differences with a p value of 0.00252. Frequency was calculated relative to most common term, lul, and a 0.01% threshold means that each term appeared either >1,700 times out of 12.8M terms in the female streamer dataset or >9,200 times out of 36.6M terms in the male streamer dataset.

In summary, yes, women streamers take more abuse than men do (though no, it’s not totally absent in chats for male streamers, either).

The methodology of our madness

We’ve selected 50 popular female and male streams to use as our samples, all of whom have at least 9,000 followers, many of whom have numbers in the 100,000s.

• To be selected, a streamer needed to be streaming in English (‘cause writing an NLP script that handles multiple languages simultaneously is not something we wanted to deal with).
• The person had to be streaming actual live gameplay, to keep the different chat contexts similar.
  • That means, no game debates, reviews, draw-alongs, or Twitch IRL streamers.
• We intentionally selected both male and female streamers playing a variety of games (shooters, real-time strategy, card, MOBA, RPG, turn-based strategy, platformer, etc).
  • We didn’t narrow it down to a particular game or genre for a number of reasons, including a desire to get as representative a swath of gamers (and their audiences) as possible.
  • There’s no real way to control for streamers switching games over the course of the test period (2 weeks), as many almost certainly did. There is no log in the chat of game played in-stream (other than the organic mention of the game name by audience members). The assumption with which we are working is that game changes in a population of widely-spread genres re-shuffled in a non-predictable way.

We then applied a list of sexist terms frequently used to demean women gamers, sexist terms used to demean male gamers, and generic insults (i.e. moron,n00b|noob,[^a-z]lame,etc).  This was necessarily somewhat subjective, so we both focus-grouped this and saw what the actual subject population used.

We then logged all the chat transcripts with irssi bots using Twitch’s API (if you’re interested in doing this yourself, Crunchprank has a very easy to follow starter guide).  Then, we dumped those to text files, removed usernames (for privacy and logistical purposes), and system messages (join messages, user counts, etc).

Next, we threw the ngram parses at them (no, not literally, though debugging kinda made us want to a few times.  CURSE YOU, PYTHON MEMORY ERRORS!).

Presto, changeo!  Term frequencies for male and female streamers, expressed as a relative percentage compared to the most common non-name term used (as mentioned above, lul), which was set as equal to 100%.  Thus, normalized_tf = 0.10 means “something used 10% as often as lul.” 

Note – terms were only included if they appeared with a normalized frequency of at least 0.01.  Anything less common than that we considered to be so rare as to fall outside of the context of the study, which was “language used by the Twitch community.” For example, “neurotic” is almost always used to describe women….but not on Twitch, as it appeared in the dataset 2 times (out of 49,335,820 words).

What’s next?

This was a test, looking at all 50 streamers of each sex as a big aggregate, over 2 weeks of data.

We’ll keep collecting transcripts until we get a month’s worth, and then we should have enough of a text dataset to compare each streamer as an individual corpus, looking at:

• Is the sexism primarily caused by a small group of bad actors (common to both datasets or independent), or is it pretty evenly spread out?
  • Breaking things down by (anonymized) username will necessitate writing a new text parsing script, but it should be doable. Basically the workflow would be something like this: Search through all files in the log directory -> Get a list of unique strings that start a line with “ – ” -> Run a loop over each of those usernames, collecting the rest of the line from matching lines on all files -> Dump result into text files named for each username.
• How terms clustered.
• Whether there was any pattern to types of sexist language.
• Studying the uniformity of sexist language.
  • An even spread of sexist language would look the same as a few asshole-magnet streamers pulling the averages up in an aggregate datatset like this, it’s impossible to differentiate without segmenting them further.

Stay tuned!

Special thanks to Alex Lehman, Ashley Rivas, and Kate W. Zimmerman for their feedback and help with this project, particularly with compiling and analyzing the test set of terms.