Tomorrow's response question repeats a similar question from last year, asking you to think critically about methodology and how criteria shape implications of research.
In his conclusion, Evan Rinquist writes that while race is a statistically significant factor in the selection of environmentally hazardous sites, it is nonetheless a small one. He goes on further to say that the small size of its impact means that race should remain only one of many criteria for policy creation. While this is an interesting conclusion that should and probably will be discussed further, let's think about how Rinquist arrived at his conclusion.
Looking at page 230 of the article, the section entitled "The Data Gathering Process," we would like you to think about how the research decisions affect meaning and outcomes.
How did he sort through those studies and how many were discarded? What did he keep? What does that say about his conclusions? How far, and to what can we apply his conclusions? (Hint: is he reporting the results of his analysis of readings, or that of original research on environmental justice?) How do Rinquist's criteria reflect the general "entry requirements" for evidence in the quantitative or scientific fields, what does it mean for the study of environmental justice if the bar for admission is set high? What does it mean if the bar is set too low?
P.S. If you have a strong statistical background, great! The article will be a much richer experience for you. If not, however, don't worry too much--and *try* not to get too bogged down in the statistical minutiae.
Subscribe to:
Post Comments (Atom)
8 comments:
Despite explaining his research process clearly and thoroughly, I think Rinquist's conclusions are not an accurate representation of reality. He bases his data entirely off research done by others and therefore presents nothing novel. I don't think his conclusions can be widely applied because his data is only as good as the research done by others. Rinquist himself states that different factors such as sources of potential environmental risk, definition of community, etc. can cause disagreements in the conclusions made by researchers. However, it doesn't appear as though he takes these factors into account when deciding on the literature he will use for his own data. It seems that he picks his sources based on their relevancy to environmental equity rather than the quality of the facts presented. While he claims to standardize the data using a random effects model, I don't feel that this method effectively standardizes the techniques and errors introduced by the authors of the initial studies.
Besides the way in which he analyzes the data, Rinquist's conclusions may be skewed based on the way he chose his sources. He originally sorts through the literature using keyword searches. This doesn't seem very 'scientific' to me, as some valuable research may not be considered only because it does not contain the specific words he searches. However, I suppose he doesn't have many options when trying to narrow down 49 relevant sources from thousands. He ended up keeping literature based on its relevance to environmental equity rather than the specific data each collected.
Rinquist's criteria seem random and pertain only to relevancy to the issue. He also selected literature from the reference sections of some studies which may skew his conclusions since studies cited in the same source are likely to contain similar data. If he were to have set the bar for admission higher, many valuable sources may have been eliminated. On the other hand, had he set the bar for admission lower, some inappropriate sources may have 'contaminated' the data.
Ringquist found peer-reviewed articles, nonpeer reviewed articles, books, government documents, private reports, dissertations and Master’s theses by searching for the keywords “environmental equity,” “environmental justice,” and “environmental racism.” His team also searched the websites of think tanks and interest groups related to environmental justice and searched for their equity reports. The Office of Environmental Justice also provided government reports on environmental equity. Finally, his team picked through the reference section of studies.
To filter down the list they compiled, Ringquist and his team analyzed documents only qualifying as “potentially relevant.” This removed newspaper and magazine articles along with other popular media sources. Then, non-analytic (descriptive), nonquantitative, non-US, and research examining a different dependent variable other than those in their research questions were discarded. They had to be analytic, using statistical data and techniques, and focused on race or class inequities in the distribution of noxious facilities, Superfund sites, or pollution levels in the US. 49 acceptable results out of thousands survived the filtering.
Ringquist states that the only reports that do not show inequities due to race used “questionable” methods, but he also found that race-based environmental inequities, while they do exist, are small. Class-based environmental racism is even less significant. Therefore, while race-based inequities are undeniable, they do not account for the majority of the problem, and therefore policy should not focus on them.
Ringquist does not gather his own statistical data, but rather uses the data of reports that make their way through his filtering system in order to arrive at his conclusion. During the filtering process, the statistic for how often his researchers agreed initially upon significance of an article was .85, suggesting that his techniques for choosing reports was subject to individual biases. While .85 is not a dreadful statistic, it still appears to me to be a significant gap between what individual people viewed as acceptable and unacceptable. How were these discrepancies resolved? The criteria seem relatively simple to me. What caused the discrepancies? Surely a group of well-trained scientists knew how to pick out a report that showed excellent statistical technique and that practiced strong analytical prowess.
Out of thousands of reports, only 49 were selected as the source of data for his conclusions. The number 49 appears awfully low in the midst of thousands of available documents. Some of these documents were also linked (gathering data from one report and also from the report’s reference list). I do not know the number out of 49 that were linked in this manner, but even a small amount out of such a small group of accepted documents could skew his results towards one particular conclusion, creating an extra variable. It would have been better to use data from entirely unrelated sources in order to keep his sample set pure.
As a supporter of the scientific method and the importance of facts rather than speculation and assumption, I do not denounce relying a great deal (although not solely) on scientific means to gather evidence for social justice realities. I do, however, think it is important to realize that scientists have throughout the ages used numbers and seemingly “scientific” methods in order to oppress people and truths. The Nazis considered their methods of measuring physical features to determine racial superiority quite scientific, for example, for it was a matter of human evolution. More recently, conservatives and the religious right have employed scientists to “disprove” global warming and evolution. The methods of science itself are not dangerous, but the fact that humans gather, interpret, and release their scientific findings can often lead to problems, as scientists are ultimately human, subject to personal biases, errors in judgment, and, of course, money. We must make sure to expand the amount of research we are willing to look at as well as we must take care to match our scientific conclusions with what we observe happening on earth. If our data does not match observable phenomena, it must be accounted for.
Unfortunately Geography 360 (statistics for geography majors) ended in May and my knowledge seems to have flown out the window. But in reading Rinquist’s meta-analysis of ej studies, he seems to have made some questionable choices regarding his methods for research.
Before addressing some confusing points in Rinquist’s methods, I must admit that he aims to raise the bar for social science research. Using quantitative data and methods will surely enrich the body of research surrounding ej issues, but he is perhaps excluding some other important academic approaches. I do agree with Christine, that whittling his sample size down to 49 is perhaps a statistically faulty move. He started with thousands of studies, and in employing his personal judgment, discarded many ‘potentially relevant sources’. This could be a shortsighted strategy when looking for answers to whether and where environmental inequities exist.
Descriptive and qualitative evidence is important because it allows for narrative and personal experience to influence a study. Rinquist is surely not exposed to environmental injustices on a daily basis (and neither am I) but that doesn’t mean that our studies should be blind and insensitive to the few people already oppressed by the system. What if one of his potential sources was a letter to the editor, written by a victim of environmental injustice, describing a destructive work environment? This type of ground up information IS valuable, since it can inform future quantitative studies. Turning this blind eye is also problematic because it is already difficult to convincingly attribute illnesses such as cancer to localized hazardous places or pollutants.
Perhaps I got slightly off track. He does raise a good point in illustrating the need for statistically substantive evidence for ej. We do not want a field based on conjecture and Oprah-esque stories, however he may have gone too far by altogether writing off qualitative research. This point can be seen in the measly 49 studies he came up with. As much as he references statisticians, his small sample size is laughable. One last idea that is lost in this article is that it is impossible to conduct a perfect study. Moreover, theoretical frameworks should be fluid and subject to revision. Perhaps this idea is important to keep in mind while reading his article, both in respect to how he addresses other peoples’ work and how we address his.
Rinquist sorted out thousands of studies based on what types of research in different types of publications such as scholarly journals, books by scholars,government documents, graduate students works,and private reports. The initial search brings him thousands of hits on environmental equity, environmental justice and environmental racism. He narrowed it down to 297 studies based on how potentially relevant they might be in his analysis. He discarded popular sources and studies that are "descriptive,nonquantitative, non-U.S. or examined a dependent variable" which is different from his research. His conclusion is based on others' research outcomes in what he collected, and he concluded that economic class was not a significant factor for environmental equity while the economnic standing is widely believed as a leading factor of environmental justice.
In terms of quantitative analysis, I think he presented a large amount of studies to make his argument. However, his method of searching the relevant studies to examine seems broad and not "scientific" because not all relevant sources contain the exact phrases.
In the process of reading summary and abstract, it does not seem clear how he defined quantity and U.S. because quantitative means large number of population studied in general or relative quantity for the population and authors can be from U.S. but the study location is non- U.S. and vice versa. It is easy to assume what he means, but it is very important to understand his definitions so that the studies can be more contextuaslized and direct.
If the bar for admission is set high, more relevan sources will be overlooked in terms of environmental justice and I believe that is what Rinquist has done. On the contrary, more inappropriate data will be included in the research to ruin upcoming study outcomes.
In reading Ringquist’s writing I became biased from the beginning. I think that his argument for showing that environmental injustice cannot be categorized as being concentrated strongly in low income areas and amongst racial and ethnic minorities was what cause my bias. I found many weaknesses in the method that he used but after reading I thought more deeply about what his intention and goal was and why he used the method he did.
Sources of potential environmental risks have been concentrated among racial and ethnic minorities and the poor for hundreds of years. These people are under-represented in the decision making processes and so it is easy for me to understand why these injustices occur. As Ringquist describes there has been a growing vocal and active concern for environmental injustices and the movement has produced many people to conduct surveys and studies on whether or not these environmental burdens are truly being distributed unfairly.
Rinquist uses meta-analysis as his method to extract a conclusion. This is the first time I have been exposed to this sort of analysis and to be blunt, I was not very fond of it. One thing interestin was how Ringquist presented multiple aspects of the method he used. He made sure to point out places where the meta-analysis method falls short of being perfect. It only extracts a generalizable conclusion is one example of this. I think that another problem is that it treats quantitative studies as individual data pieces. This would cause the results to eliminate many groups of important information. I found it interesting how scholars in social science disciplines “have been slow to adopt the technique.” This seems explicable to me because it can’t deal with the multiple variables of the social sciences. Meta-analysis also has a problem with identifying causes of environmental inequalities. And I still don’t think I understand how it CAN establish whether environmental inequalities exist and point out where (race and class wise) and tell the magnitude without being able to identify the cause. It seems a weak method to me because it eliminating so much information. But, I understand why one would need to do this. It would be to please the people who make the decisions…because they like numbers and charts and methods and analysis that they can see a concrete conclusion.
Yes, Ringquist presented his findings well because he pointed out flaws and therefore filled holes that readers may have used to attack his argument. But, I still do not agree with the idea of the meta-analysis technique for finding how environmental injustices affect people of poverty and/ or are racial/ethnic minorities.
Using computer searches of books, articles, dissertations and thesis’, Rinquist and his group sorted through databases to first discover sources. Next the group compiled a list of think tanks and interest groups with interest in environmental justice and policy and searched the websites of the organizations. Third, all of the state environmental justice commissions listed by the group were contacted and asked to submit links to government reports. Finally, they employed the ancestry method and identified potentially relevant studies in the reference section of studies obtained traditionally. Out of the information, first they sorted through potentially relevant studies, which were identified through bibliographic references. They took the catholic approach by excluding sources from popular outlets, and made it into a search result of keywords. They identified 297 potentially relevant studies and assigned each an ID. Potentially relevant studies were excluded if they were nonanalytic, nonquantitative, non-U.S., or examined a dependent variable other than those in their research questions. Relevant studies were those that were analytic, used statistical techniques, focused on racial or class inequalities, superfund sites, or pollution levels in the U.S. Finally they took a random sample of 60 of the relevant studies. From there, the team discerned 49 acceptable sources. The reliability was .85. His conclusions were reliable, but I think taking a random sample is a very irresponsible way to do research. When there is viable information that could potentially prove a point, it’s reckless to abandon it. All that being said, the relevancy tested to be .85, which could be good enough for the data collected, and if that is the case, it still proved that economic status is not a major factor in environmental hazard.
But this is where I have my biggest problem with his findings. Reliability is a percentage, but an uncertain one. Ringquist’s team used meta-analysis which weights the size (which to me in this context means importance) by its associated uncertainty. In addition he says that this testing is not used in the field of social science, possibly because this approach may not fit. The thing with percentages is that people can manipulate data to prove their point. Ringquist only uses other people’s data, which do not prove his point but rather their own, to prove his point. Through the analysis approach he uses, this may actually translate, but I question to what degree it translates to make the data 85% reliable, if none of the data explicitly proves what they are trying to. In addition to that, I am very certain, due to the proximately of environmental hazards to people of low socio-economic class and race, that it is environmental racism, what I would have liked him to do is to disprove cases that portray environmental racism, instead of prove the opposite.
“Assessing Evidence of Environmental Inequities” by Ringquist clarified how important research decisions are. The publications that are chosen to be included, the research methods to find the publications, and how each individual publication and groups of publications will be weighed wholly affects the meaning and outcome of the research and has the ability to bias the results very easily. The kind of materials the researcher decides place importance on and how s/he weighs them compared with other materials will change the outcome and meaning of what is being researched completely, which of course will make a biased finding. Ringquist talked about how he changed how much specific research would count in the end outcome of the study by the “effect size” of it, thereby creating a system of researching that was based up his own ideas. Six types of publications were identified that might contain studies for inclusion in the meta-analysis: peer reviewed articles; nonpeer-reviewed articles; books and book chapters; government documents; private reports issued by interest groups or think tanks; Ph. D. dissertations and Master’s theses. The use of meta-analysis which are “particularly well-suited for extracting generalizable conclusions from contradictory studies . . . The meta-analysis extracts an “effect size” from each study weights the effect size by its associated uncertainty, and then aggregates the results across all studies on a particular topic” was the research technique uses and appropriated. (I personally need to write down this definition just to try and understand it again after reading the article.) The definition to a non-statistically educated person does not make a whole lot of sense unless it is broken down.
The article was very hard for me to read because it used a lot of terminology that I am not familiar with and did not use any anecdotes whatsoever. Professor Marquez said something about how people think differently and understand and connect to arguments better if they are presented in the way you think in his lecture on Wednesday June 4th. A paraphrased version of one remark of his from lecture was, some people think quantitatively and others like Barack Obama’s poetic speeches. I do not think quantitatively and this article was very quantitative. It was difficult for me to follow the various analyses and tables, though the conclusion and abstract and a few other parts were clear. It seems the author and I think in two extremely different manners and if we were on a team we would do well because you need the hard numbers as well as the stories to reach everyone as I think Yeri was trying to point out yesterday in class as well, if I understood him correctly.
Ringquist seemed to be very factual in his analysis and did not seem to let too much bias into the results, and even with all of his conservative measures he still included comments about on recent research findings, which show poor and minority neighborhoods as being more likely to contain commercial hazardous waste facilities though he also makes an odd comment “Given their foucs on current environmental inequities, neither the original studies nor the meta-analysis can tell us much about the cause of environmental inequities if they exist”, which I believe shows that he does believe environmental inequities do in fact exist, and he also cited himself several times in his article.
The bar for entry requirements is set very high and since it is hard to document with numbers environmental racism it means that it may be very hard to fight environmental abuse for people of color and low income since many who could make the legal changes to help those affected can only think in numbers.
Post a Comment