Le biais de confirmation: The story of the Bilingual Advantage 

The newest salvo in the psychology’s “reproducibility crisis” is not in social psychology, but is hitting the field of psycholinguistics. In this case, the evidence is mounting that the so called “bilingualism advantage” may not be an advantage after all. Worse, it may something like the Mozart Effect for psycholinguistics…That is, an effect that is plausible and desirable enough (and marketable) that we all believe it and ignore reputable counter evidence.

Full disclose, our children have attended a French Immersion school, and as we live in a country with two official languages, I think it’s important to know some of both. But I’m not bilingual myself (6 years of German in high school and university, but I no longer speak it). So I like the idea of a second language. And I like the idea of the bilingual advantage too. I’ve assumed that this advantage is present and measurable, but now I’m not so sure. This controversy is worth paying attention to.

The Bilingual Advantage

The story goes like this. People who speak two languages fluently are constantly switching between them. They have to inhibit one language in order to respond in the other. And because switching and inhibition are two of most well-known and well-studied aspects of the cognitive control system known as the executive functions, it’s assumed that bilinguals would be especially adept at these behaviours. And if they are good and switching and inhibiting within language, they may have a general executive functioning advantage for behaviours as well.

Ellen Bialystok at York University and others have investigated this claim and have produced quite a lot of data in favour of the idea that general executive functioning abilities are superior in bilinguals relative to English speaking persons. The advantages might also persist into old age and the may guard against cognitive decline in aging. Dr. Bialystock’s work is extensive has relied on many different measures and she’s arguably one of the towing figures in psycholinguistics.

Does this work Generalize?

An article in Feb 2016 in The Atlantic has suggested that recent attempts to replicate and generalize some of this work have not been successful. Specifically, the work of  Kenneth Paap, at San Francisco State has argued that there is no general advantage for bilinguals and that any advantages are either very local (confined to language) or are artifacts of small sample size studies with idiosyncratic groups. Systematic attempts to replicate the work have not been successful. Reviews of published studies found evidence of publication bias. And other psychologists have found the same thing. In other words, according to Paap, whatever advantages these groups might have shown in early studies, they can’t really be attributed to bilingualism.

The Battle Lines

By all accounts, this is a showdown of epic proportions. According to the Atlantic article, Paap has thrown the gauntlet down and said  (paraphrased) “Let’s work together, register the studies, collect good data, and get to the bottom of it.” Even a co-author of one of the most well-cited bilingualism advantage papers is now questioning the work. My colleague at Western, J. Bruce Morton, is quoted as saying:

“If people were really committed to getting to the bottom of this, we’d get together, pool our resources, study it, and that would be the end of the issue. The fact that people won’t do that suggest to me that there are those who are profiting from either perpetuating the myth or the criticism.”

But proponents of the advantage are not interested in this, suggesting that the Paap and others are not properly controlling their work and also pointing to their recent work with brain imaging (which gets away from the less than idea executive functioning tasks but also could fall prey to the Seductive allure of Neuroscience…which is another topic for another day).

This is, I think, a real scientific controversy. I think we should get to the bottom of it. If the advantage is robust and general, then it’s going to show up in these newer studies. If it’s not, then it becomes an outmoded idea (like so many in psychology and in science). That’s progress. There is the risk that inherent appeal of the advantage will allow it to persist even if the science does not back it it, and that’s problematic.

Advertisements

Cognitive Bias and Guns in America

UPDATE Oct 3, 2017: I wrote this almost exactly two years ago, following an earlier mass shooting. I think it still applies and helps to explain why we have trouble even talking about this. 

I posed the following question this week (Oct. of 2015)  to the students in my 3rd year Psychology of Thinking class.

“How many of you think that the US is a dangerous place to visit?”

About 70% of the students raised their hands. This is surprising to me because although I live and work in Canada, and I’m a Canadian citizen, I grew up in the US; my family still lives there and I think it’s very safe place to visit. Most students justified their answer by referring to school shootings, gun violence, and problems with American police. So although none of these students had even actually encountered violence in the US, they were thinking about it because it has been in the news.

Cognitive Bias

This is a clear example of a cognitive bias known as the Availability Heuristic. The idea, originally proposed in the early 1970s by Daniel Kahneman and Amos Tversky is that people generally make judgments and decisions on the basis of relevant memories that they retrieve and that are thus available at the time that the assessment or judgement is made. In other words, when you make a judgment about a likelihood of occurrence, you search your memory and make your decision on the basis of the available evidence. Most of the time, this heuristic produces useful and correct evidence. But in other cases, the available evidence may not correspond exactly to evidence in the world. We typically overestimate the likelihood of shark attacks and random violence because these low probability events are highly salient and available in memory.

Another cognitive bias (also from Kahneman and Tversky) is known as the Representativeness Heuristic. This is the general tendency to treat individuals as representative of the the entire category. For example, if I have a concept of American gun owners as being violent (based on what I’ve read or seen in the news), I might infer that each individual American is a violent gun owner. I’d be making a generalization or a stereotype and this can lead to bias. As with Availability, the Representativeness Heuristic arrises out of the natural tendency of humans to generalize information. Most of the time, this heuristic produces useful and correct evidence. But in other cases, the representative evidence may not correspond exactly to individual evidences in the world.

The Gun Debate in the US

I’ve been thinking about this a great deal as the US engages in yet another debate about gun violence and gun control. It’s been reported widely that the US has the highest rate of private gun ownership in the world, and also has an extraordinary rate of gun violence relative to other counties. These are facts. Of course, we all know that “correlation does not equal causation” but many strong correlations often derive from a causal link.

So why to do we continue to argue about this? One problem that I rarely see being discussed is that many of us have limited experience with guns and/or violence and have to rely on what we know from memory and from external sources and we’re susceptible to cognitive biases as a result.

For example, I’m not a gun owner any more, but many of my friends and family are, and these people are some of the most responsible gun owners I know. They own firearms for sport and also for personal protection and in some cases, even run successful training courses for people to learn about gun safety. From the perspective of a responsible and passionate gun owner, it’s quite true that the problem is not guns but the bad people who use them to kill. After all, if you are safe with guns and all your friends and family are too, then you base your judgements on the available evidence: gun owners are safe and so gun violence is not a problem of guns and their owners, but a problem of criminals with bad intentions.

But what about non gun owners?  Although I do not own a gun, I feel safe at home. My personal freedoms are not infringed and I recognize that illegal guns are the problem. And so I generalize this experience and I may have difficulty understanding why people would need a gun in the first place whether for personal protection or for a vaguely defined “protection from tyranny”. From our perspective it’s far more sensible to focus on reducing the number of guns. After all we don’t have one, we don’t believe we need one, so we generalize to assume that anyone who owns firearms might be suspect or irrationally fearful.

In each case, we are relying on cognitive biases to infer things about others and about guns. These things and inferences may be stifling the debate and interfering with our ability to find a solution

How do we overcome this?

It’s not easy to overcome a bias, because these cognitive heuristics are deeply engrained and indeed arise as a necessary function of how the mind operates. They are adaptive and useful. But occasionally we need to override a bias.

Here are some proposals, but each involves taking the perspective of someone on the other side of this debate.

  1. Those of us on the left of the debate (liberals, proponents of gun regulations) should try to recognize that nearly all gun enthusiasts are safe, law abiding people who are responsible with their guns. Seen through their eyes, the problem is with irresponsible owners. An attempt to place restrictions on their legally guns activates another cognitive bias known as the endowment effect  in which people place high value on that which they already possess, and the prospect of losing this is aversive.
  2. Those on the right (gun owners) should consider the debate from the perspective of non gun owners and might also consider that proposals to regulate firearms are not attempt to seize or ban guns but rather attempts to address one aspect of the problem: the sheer number of guns in the US, all of which could potentially be used for illegal purposes. We’re not trying to ban guns, but rather to regulate the and encourage greater responsibility in their use.

I think these things are important to deal with. The US really does have a problem with gun violence. It’s disproportionally high. Solutions to this problem must recognize the reality of the large number of guns, the perspective of non gun owners, and the perspectives of gun owners. We’re not going to do this by simply arguing for one side. We’re only going to do this by first recognizing these cognitive biases and them attempting to overcome them in ways that search for common ground.

As always: comments are welcome.

The Way the Music Died

I almost never listen to music any more. That’s a sad state of affairs for anyone, but strikes me as a nearly complete disconnect from my former self.

I’m an amateur musician.

I’ve played music since I was about 10 or so..I’m 45 now. First the trombone, then pipe organ lessons at age 12 (two things that failed to endear me to girls), piano, music theory in high school (including a good score on the AP exam), glee club, college choir. Not to mention electric bass beginning around 15 and continuing to this day. Years of playing rock and roll, blues, modern rock (covers and original songs) in bars and clubs.

I used to love listening.

At various times I’ve had decent CD collection, LPs, cassettes, thousands of digital files. I packed up CDs when I moved from Lewisburg to Buffalo. Merged the collection with my wife’s cooler and better collection. We packed it to Champaign-Urbana and then here to London. Ripped the entire thing to MP3, and then again at a better rate. Have consumed walkman, portable CD players, original iPods, iPhones and Android devices. Many of these devices still litter the bottoms of drawers and boxes in our house.

But where did the love go?

But in the last few years I’ve just nearly stopped listening to music. Why? I suppose there were practical reasons. Could not really blast The Pixies when a 2 year old was taking a nap. And I’ve taken on other hobbies, responsibilities, wasting time on FaceBook. And I’ve become a lover of silence and stillness.

But I miss…really miss… the pleasure of just listening to music.

What’s bothering me is that I don’t know what to listen to any more. Heck, I don’t even know what genre (s) of music I like anymore. I think it was the streaming that killed it for me. Streaming is the main way to listen. But just like the mp3 nearly killed CD and LP’s, Streaming has killed digital downloads. There is just so much content on streaming services. I have Google’s Play Unlimited, but we’ve used Grooveshark, Songza, and Spotify and a lot of YouTube.

So I never even know where to begin. Recorded music was once a distinct and quantifiable pleasure for me and now it seems like an infinite stream of background.

There is no investment in this for me. None. It’s always on. It never ends. The recommendations are either too familiar or too foreign. I feel like it’s a lifetime of trying and sampling and no room for commitment. When i do listen, I just around haphazardly. It feels empty. It’s killing my love of music because there is really nothing to love. I fully admit that some of this is my problem. I’m not using the streaming services the way I should. I’m not following people and listening to their recommendations and playlists.

I’m not ready to let it go.

So I’m trying to figure out how to get back into music listening, but that’s so hard to do. I bought a used turntable a few years ago, which is fun. But that means I need to build up my record collection. I’m considering a new/used CD player and starting to build that up again. But that has its downsides too: I don’t really want boxes of CDs taking up space. Perhaps I should just embrace a streaming service and just set aside time for real listening. Do others feel this way? Do you love streaming? Has it enhanced your music listening pleasure or has it detracted.

Redesigning my Graduate Seminar

I’m writing this to crowdsource and to get advice on course design, so this might not be the most interesting blog entry I’ve ever written. But if this is the kind of thing that intrigues you…read on!

I teach a graduate seminar on cognition every two years. It’s one of the required courses for our cognition graduate students (Master’s and PhD) and covers the fundamental issues in the field, and also to help prepare students for the PhD exams. I usually have a few students from other areas in psychology (e.g. social psychology) as well as student from other departments (philosophy, neuroscience, and business). I’ve run this class since 2007, and I’m thinking of overhauling some of the topics for the Fall 2015 session. An example of the most recent syllabus can be found HERE.

The basic course

The official description of the course as it stands now is as follows.

“This course aims to provide graduate students with exposure to classic and current research in cognitive psychology. We will read and discuss articles on the major topics in the field, including high-level perception, categorization, attention, working memory, knowledge, language, and thought. The readings will encompass theoretical approaches, behavioural research, computational modelling, and cognitive neuroscience. Meetings will follow a seminar format, in which students will discuss the readings for each class. To frame the discussion for each meeting, the instructor will provide background and any needed tutorials. Marks will be based on participation and written work.”

The class has a midterm, a final paper, and discussion papers. I cover topics like:

  • Viewpoint Independent vs. Viewpoint Dependent Theories of Object Recognition
  • Theories of Working Memory”
  • Inductive Reasoning: Similarity-based or Theory-based?
  • “Categorization: Prototypes vs. Exemplar models”

This is all pretty standard graduate seminar material. But I’m thinking of updating some topics and in particular  updating the way the course is run. I’m not sure the standard format is the best, and the last time I ran a seminar (not this one, but a class on “Concepts and Categories” ). I felt that it was not working well. Discussion did not flow, I talked too much, there was a lack of enthusiasm. This was not due to the students, but I think was a result of my not leading it well, and the limitations of the standard seminar format.

A new format

I’m planning to keep some the topics, but add in new newer ones that could be challenging or debate-worthy. For example

  • The Dual-Process Approach of Thinking
  • What are Mental Representations and Are They Needed for a Theory of Cognition?
  • Cognitive Heuristics: Helpful or Harmful”.
  • Is Vision Strictly Visual: Vision as Action, Evidence from Blind Echolocation, and Visual imagery

Each class would have around 4-5 papers assigned which would be a mix of classic, standard papers, and new perspectives on that issue. If possible, I will assign readings that cover two sides to an issue. In the past, there are usually 10-15 people in people in this seminar, and it meets for three hours, typically two 1:15 sections and  short break.

Here are four possible ways to run a class. I’ve usually used  the first two…Sometimes it works well, sometimes it works less well.

1. The Standard Seminar. Each class is a discussion of the central issues. I provide 3 or 4 central questions (or maybe a brief outline) that students can prepare for. Rather than discuss each paper in detail, we will try to answer the central questions but will use the assigned readings as a guide.

2. Student-led Discussions. Each class can have one student assigned to be the discussion leader. Alternatively, I act as discussion leader, but there is one student assigned to present each paper.

3. The Debate Team. Each class is framed around a debate and there will be two teams. Each team consists of a leader and two panelists. They will face off and the remaining students will asses and judge the winner and provide feedback. A three hour class could have one or more debates. Throughout the course of the class, there will be several opportunities to be on an debate team.

4. The Shark Tank. This format is like a debate, but one team is selling the idea and the other team is an antagonist. Students make a 10 min presentation of the paper, theory, or idea and then defend the ideas in the paper, and the the sharks question and critique the ideas. Sharks decide to accept the idea and invest, or pass. The rest of the class participates by assessing the performance of the sharks an the presenters.

Ideas?

So if you were taking a course like this, which if these formats would you enjoy most? Which format do you think might promote the best mastery of the material, the best engagement? Are there other ways to run a graduate seminar that I have not tried?

Maybe I should use a mix of formats? Some classes will be standard discussion while others will be debates?

I’d welcome and appreciate any suggestions, critiques, and ideas.

Almost no one reads my work. Should I care?

I recently read an article that has been going around social media in which the authors argue that basically no one is reading academic journals. They argue that in order to be heard, and in order to shape policy, professors and academics should be writing Op-Eds.

The article, which I’ve linked to here,  should be read with a few caveats. First of all, the authors suggest that the average academic paper is read in total by about 10 people. They provide no evidence or information about how they arrived at that estimate. Second, they are writing from the standpoint of social science and political science. In other words, the results may not apply to other disciplines. That said, I believe there are many reasons to take their idea seriously.

There are too many articles published every year.

There are so many scientific and academic journals operating right now. For example, the popular journal PLoS ONE  published 31,500  articles in 2013… That’s 86 articles a day.  In 2014, the published even more….33,000 articles. Only one of them was from my lab.  Now I happen to think that this particular article was a really good paper. It was based on my student Rachel’s master’s thesis. But it’s only one of over 30,000 articles that year. According to the statistics on their own site, there were about 1400 views of our article. So far it’s been cited twice.  Is that good? Is that enough?  Should I care? After all, it’s only one paper of many that I have published in the last few years

This is only the tip of the iceberg. As I said, this is one journal. There are other large journals like PLoSONE.  And there are many, many smaller journals with limited output. But still, it’s estimated that this year alone there will be over well 2 million articles published.  Even if you assume that within your own field, it’s only a few thousand every year, finding the ones that matter can still be a problem. If you use Google scholar (and I do) to research, you may have noticed that it  it tends to place heavily cited articles at the top of the search. This is good, because it gives you a sense of which articles have had the most impact in the field. This is bad because the first thing you see is the same article that everyone else has cited for the last 20 years. Unless you take the time to adjust your search, you are not going to see any of the new work.

And as if this isn’t problem enough, there have been widely reported problems with the academic publishing world. For example some journals have even had to withdraw articles, many articles, when it was revealed that they were entirely computer-generated gibberish.  There are also hundreds and hundreds of so-called “predatory” journals in which the peer review is nonexistent, standards for publication are very low, and the journals operate solely to make money publishing papers that otherwise wouldn’t be published. You can see a list of these predatory journals here.  Even journals published by well-known companies have had difficulty recently. In some cases, editors have been accused of accepting articles with little or no to review.

Why do we do this?

I cannot speak for other academics, but within my field and for me, the reason is simple. It’s my job. As a professor in a large research institution, part of my job is to carry out scientific research, and publish the scientific research in peer-reviewed journals. Publishing papers in peer-reviewed journals was necessary for me to obtain tenure. It is necessary for me to be able to compete for federal research dollars. Other forms of communication can help in terms of getting the message out, but as it stands now, publishing a popular article, a textbook, a Op-Ed, a newspaper article, or even a popular blog essentially does not count. I might as well be doing that on my own time. Which, I suppose I am.

So in essence, we have a designed and embraced system that rewards publications in one format and does not reward publications in other formats. Unfortunately, the format that is rewarded is one in which almost no one outside of the immediate field will ever read.

What do we do about this?

I am not entirely sure what to do about this, but I do believe that it is a real problem. I’m not suggesting that scientists and academics abandon publishing in academic journals. In fact, I still think that’s exactly where primary research belongs. As long as the peer review is being carried out properly, editors are behaving properly, and editorial standards are high, this is exactly where you want your best work to appear. I also don’t want it to be the case that scientists and academics begin pursuing popular media at the expense of academic publishing.

What I would like to see, however, is an appropriate balance. It might be time for internal performance review committees and promotion and tenure committees to broaden the scope of what counts as scientific, academic, and scholarly contributions. This will provide some incentive for researchers to publicize their work. At the very least, researchers should not be penalized  for attempting to engage the public in their research. A successful research program is one that publishes in different outlets and for different audiences. We do this in my department, we work with the local public library, for example, to engage in popular science topics in psychology.  Our research communication office works long and hard to publicize and promote research. However, much of this is still considered to be secondary.

Another possibility, one that is suggested by the editorial staff at PLoSONE,  is for individual researchers to publicize their own work.  Researchers should share and Tweet their research as well as others.  And there are other formats, Google+, for example hosts a large scientific community that publicizes research, shares research, and even organizes virtual conferences. I’ve taken part in some of these, and they can be an effective way to share your work.

In the end, I wonder if we should all slow down, work more carefully, and think long and hard about the quality of our research versus the quantity of our publication output.  Otherwise, I think there is a real concern that the signal will be completely drowned out by the noise.

Grade Inflation at the University Level

I probably give out too many As. I am aware of this, so I may be part of the problem of grade inflation. Grade inflation has been a complaint in universities probably as long as there have been grades and as long as there have been universities.

Harvard students receive mostly As.

But the issue has been in the news recently. For example, a recent story asserted that the most frequent grade (e.i. the modal grade) at Harvard was an A. That seems a bit much. If Harvard is generally regarded as of the world’s best universities, you would think they would be able to asses their students on a better range. A great Harvard undergrad should be a rare thing, and should be much better than the average Harvard undergrad. Evidently, all Harvard undergrads are great.

One long time faculty member, says that “in recent years, he himself has taken to giving students two grades: one that shows up on their transcript and one he believes they actually deserve….“I didn’t want my students to be punished by being the only ones to suffer for getting an accurate grade,”

In this way, students know what their true grade is, but they also get a Harvard grade that will be an A so that they look good and that Harvard looks good. It’s not just Harvard, of course. This website, gradeinflation.com, lays out all details. Grades are going up everywhere…But student performance may not be.

The University is business and As are what we make.

From my perspective as a university professor, I see the pressure from all sides, and I think the primary motivating force is the degree to which universities have heavily embraced a consumer-driven model. An article The Atlantic this week got me thinking about it even more. The article points out, we (university) benefit when more students are doing well and earning scholarships. One way to make sure they can earn scholarships is to keep the grades high. It is to our benefit to have more students earning awards and scholarships.

In other words, students with As bring in money. Students with Cs do not. But this suggests that real performance assessment and knowledge mastery is subservient to cash inflow. I’m probably not the only one who feels that suggestion is true.

And of course, students, realizing they are the consumer, sort of expect a good grade for what they pay for. They get the message we are sending. Grades matter more than knowledge acquisition. Money matters more than knowledge. If they pay their tuition and fees on time, they kind of expect a good grade in return. They will occasional cheat to obtain these grades. In this context, cheating is economically rational, albeit unethical.

Is there a better system?

I am not sure what to do about this. I’m pretty sure that my giving out more Cs is not the answer, unless all universities did this. I wonder if we really even need grades? Perhaps a better system would be a simple pass/fail? Or Fail/Pass/Exceed (three way). This would suggest that students have mastered the objectives in the course and we (the University) can confidently stand behind our degree programs and say that our graduates have acquired the requisite knowledge. Is that not our mission? Does it matter to an employer if a student received an A or a B in French? Can they even use that as a metric when A is the modal grade? The employer needs to know that the student mastered the objectives for a French class and can speak French. Of course, this means that it might be tricky for graduate and professional schools to determine admission. How will medical schools know who admit if they do not have a list of students with As? Though if most students are earning As, it renders moot that point.

In the end, students, faculty, and university administrators are all partially responsible for the problem, and there is no clear solution. And lurking behind it, as is so often the case, is money.

A Computer Science Approach to Linguistic Archeology and Forensic Science

Last week (Sept 2014),  I heard a story on NPR’s morning edition that really got me thinking…(side note, I’m in Ontario so there is no NPR but my favourite station is WKSU via TuneIn radio on my smart phone). It was a short story, but I thought it was one the most interesting I’ve heard in last few months, and it got me thinking about how computer science has been used to understand natural language cognition.

Linguistic Archeology

Here is a link to the actual story (with transcript). MIT computer scientist Boris Katz realized that when people learn English as second language, they make certain errors that are a function of their native language (e.g. native Russian speakers leave out articles in English). This is not a novel finding, people have known this. Katz, by the way, is one of many scientists that worked with Watson, the IBM computer that competed on jeopardy

Katz trained a computer model to learn from samples of English text productions such that it could detect the writer’s native language based on errors in their written English text. But the model also learned to determine similarities among other native languages. The model discovered, based on errors in English, that Polish and Russian have historical overlap. In short, the model was able to determinethe well know linguistic family tree among many natural languages.

The next step is to use the model to uncover new things about dying or languages. As Katz says

But if those dying languages have left traces in the brains of some of those speakers and those traces show up in the mistakes those speakers make when they’re speaking and writing in English, we can use the errors to learn something about those disappearing languages.”

Computational Linguistic Forensics

This is only one example. Another one that fascinated me was the work of Ian Lancashire, an English professor at the University of Toronto and Graeme Hirst, a professor in the computer science department. The noticed that the output of Agatha Christie—she wrote around 80 novels, and many short stories— declined in quality in her later years. That itself is not surprising, but they thought there was a pattern. After digitizing her work, they analyzed the technical quality of her output and found richness of her vocabulary fell by one-fifth between the earliest two works and the final two works. That, and other patterns, are more consistent with Alzheimer’s than normal aging. In short, they are tentatively diagnosing Christie with Alzheimer disease, based on her written work. You can read a summary HERE and you can read the actual paper HERE.  It’s really cool work.

Text Analysis at Large

I think this work is really fascinating and exciting. It highlights just how much can be understood via text analysis. Some of the this is already commonplace. We educators rely on software to detect plagiarism. Facebook and Google are using these tools as well. One assumes that the NSA might be able to rely on many of these same ideas to infer and predict information and characteristics about the author of some set of written statements. And if a computer can detect a person’s linguistic origin from English textual errors, I’d imagine it can be trained to mimic the same effects and produce English that looks like  it was written by a native speaker of another language…but was not. That’s slightly unnerving…