Internet searches can measure racism, depression, child abuse and hateful mobs

Want to know more about secrets of people? Just analyse their Internet searches, says a book by Seth Stephens-Davidowitz, who has used Google data to measure racism, self-induced abortion, depression, child abuse and hateful mobs.

Internet searches also have lots of information that is missed by the polls that can be helpful in understanding, among many other subjects, an election, he says.

"Everybody Lies: What the Internet Can Tell Us About Who We Really Are", published by Bloomsbury, explores the secrets embedded in Internet searches.

Everybody lies, to friends, lovers, doctors, pollsters - and to themselves. In Internet searches, however, people confess their secrets - about sexless marriages, mental health problems, even racist views.

Seth, an economist and a former Google data scientist, shows that this could just be the most important dataset ever collected.

He explains how we can use information to change our culture, and the questions we're afraid to ask that might be essential to our health - both emotional and physical.

On polling and voting, he writes, "There is information on who will actually turn out to vote. More than half of citizens who don't vote tell surveys immediately before an election that they intend to, skewing our estimation of turnout, whereas Google searches for 'how to vote' or 'where to vote' weeks before an election can accurately predict which parts of the country are going to have a big showing at the polls."

Seth says there might even be information on who they will vote for.

"Can we really predict which candidate people will vote for just based on what they search? Clearly, we can't just study which candidates are searched for most frequently. Many people search for a candidate because they love him. A similar number of people search for a candidate because they hate him," he says.

A large percentage of election-related searches contain queries with both candidates' names, he says.

"During the 2016 US election between Donald Trump and Hillary Clinton, some people searched for 'Trump Clinton polls'. Others looked for highlights from the 'Clinton Trump debate'. In fact, 12 per cent of search queries with 'Trump' also included the word 'Clinton'. More than one-quarter of search queries with 'Clinton' also included the word 'Trump'," he writes.

Seth says he also found that these seemingly neutral searches may actually give some clues to which candidate a person supports.

How?

"The order in which the candidates appear. Our research suggests that a person is significantly more likely to put the candidate they support first in a search that includes both candidates' names.

"In the previous three elections, the candidate who appeared first in more searches received the most votes. More interesting, the order the candidates were searched was predictive of which way a particular state would go. The order in which candidates are searched also seems to contain information that the polls can miss," he writes.

So did Google predict Trump?

"Well, we still have a lot of work to do - and I'll have to be joined by lots more researchers - before we know how best to use Google data to predict election results. This is a new science, and we only have a few elections for which this data exists," Seth says.

"I am certainly not saying we are at the point - or ever will be at the point - where we can throw out public opinion polls completely as a tool for helping us predict elections.

But there were definitely portents, at many points, on the Internet that Trump might do better than the polls were predicting," he adds.

During the general election, there were clues that the electorate might be a favourable one for Trump, he says.

"Black Americans told polls they would turn out in large numbers to oppose Trump. But Google searches for information on voting in heavily black areas were way down. On election day, Clinton would be hurt by low black turnout. There were even signs that supposedly undecided voters were going Trump's way," he writes.

Seth says he has spent just about every day of the past four years analysing Google data. This included a stint as a data scientist at Google, which hired him after learning about his racism research. And he continues to explore this data.

"The revelations have kept coming. Mental illness; human sexuality; child abuse; abortion; advertising; religion; health. Not exactly small topics, and this dataset, which didn't exist a couple of decades ago, offered surprising new perspectives on all of them," he says.