One of the more difficult tasks set for a Masters and especially PhD student is to identify limitations in the literature under review. Postgraduate students are required not only to “tell” (repeat what an author/s say) but to critically engage the literature being reviewed. This work typically appears in a separate chapter in the thesis or dissertation under the title Literature Review. To help you, here are 10 typical examples often used to identify limitations in a particular body of literature—with a caveat—using education examples.
1. The sample size is too small and therefore not representative of the phenomenon being studied
This is fine but be aware of the fact that a single case study (e.g. the leadership practices of a non-traditional principal) is completely acceptable in a qualitative study. Here depth makes up for spread (representivity) for the purpose of the study is different e.g. an ethnographic case study of one principal.
2. The context is limiting and therefore not representative of all
This too is acceptable e.g. a study of subject competency levels of 10 science teachers in rural Limpopo province. However, always specific WHAT it is about the context that is limiting e.g. nothing about science teachers in urban areas. A common mistake is to say ‘that study was done in America’ but that is not helpful in itself. What is it about the American context that might not be applicable in the Southern African context?
3. The study is dated as in out-of-date e.g. 1985
This is important as a criticism except in the case of historical studies or in reference to a classical piece of work that set the standard for understanding a particular problem. For example, Michael Apple’s 1979 Ideology and Curriculum might be dated but it is the landmark criticism in the politics of curriculum and often deserves referencing as a launching pad for more recent studies.
4. The period of observation was too short
This is valuable especially in the qualitative study of classrooms where the researcher visits a history class of 10 teachers once and then jumps to make major findings and conclusions from a 40-minute lesson per teacher. This is a common problem with qualitative research in education in South Africa—the lack of extended engagement in the field.
5. The theoretical framework or theory is limiting
A researcher might be using a particular theory to explain events e.g. a behaviorist account of learner discipline where you believe that a constructivist account offers more insight into why learners misbehave in classrooms. The onus is still on you to explain why the rival theory is ‘better’ than the one used.
6. The methodological approach is inappropriate for the question posed
A study uses a self-reporting questionnaire completed by teachers to determine their competency levels in mathematics teaching. You could argue that direct observation of actual teaching is a much more direct measure of teaching competency in mathematics since teachers might overestimate their own levels of competency in the subject (by the way, there is research to back you up with such a claim)
7. The research question or instrument(s) is biased
A study that asks ‘why teachers are incompetent’ or ‘the students are lazy’ already assumes the fact ahead of the inquiry itself. The research question can and should be posed in an open-ended manner to allow for more than one outcome. In the controversial SU study on Coloured women’s cognitive abilities and health styles it was found that the measuring instrument used was found in other studies to be flawed. And the study of code-switching in a Grade 6 language classroom would clearly show bias if the researcher was competent in English alone—unless, of course, this deficiency is remedied in the study design.
8. The study does not differentiate between subjects or contexts
A report might give the results of a study for academic performance in Grade 12 economics in the National Senior Certificate, which is fine, but does not differentiate between the results of the former white schools and those of black schools thereby concealing variable performance and, for the sake of reform, where exactly added support might be needed.
9. The literature reviewed does not cover the subject at all or from a particular perspective
The conclusion that ‘there is no research on topic X’ is a common one among students and often wrong. The fact that you have not yet found literature on topic X does not mean that such references do not exist. However, there are instances in which the case can and should be made that topic X appears to be understudied given the literature reviewed. Hint: always check with your supervisor and other experts whether the ‘not yet researched’ claim is justified. More plausible are studies which critique a dominant perspective on a subject and offer a new lens on the same issue (see point #5) above.
10. The study is based largely on opinion and uses primarily secondary sources
This is a very valid criticism of many articles posing as research studies where the evidence is often anecdotal (not ‘thick descriptions’ as anthropologists like to point out) and drawing on other opinion pieces rather than solid research on the topic.
REMEMBER: THESE NOTES ARE NOT ABOUT HOW TO IDENTIFY LIMITATIONS IN LITERATURE REVIEWED FOR YOUR STUDY. IT IS NOT ABOUT LISTING THE LIMITATIONS OF YOUR STUDY IN THE COURSE OF WRITING UP A RESEARCH PROPOSAL OR THE THESIS ITSELF, EVEN THOUGH THIS COULD HELP YOU DEFEND SELF-IDENTIFIED LIMITATIONS OF YOUR RESEARCH.
