What makes a good/tight research study?
The term “research,” however, is sometimes loosely used to mean the way data is collected. Politicians often say they formulate policy based on research; news agencies often tell us of their research findings; while marketers highlight the results of their research findings to make consumers believe in the products they are selling. However, all these only indicate the wide range of meanings of the term “research” in everyday usage. From the perspective of research methodology, the everyday uses of the term do not relay the true meaning of the word. This is where you must really be careful not to jump to conclusions about the findings.
Significance of the term ‘significance’
One thing to keep in mind when looking at ANY research in educational domain is to understand what the term ‘significant’ means. In general, findings are reviewed in terms of a 5% error term for educational practices. What is deemed to be ‘significant’ is that the results have an error term (possibility of being wrong) of less than 5 out of 100. This is all about risk/reward and significance in real world terms. What .05 error means in reality is that one out of every 20 examples will not fit the findings. So, if you were building a bridge, for example, if you were to allow a safety error of .05, then one out of every 20 cars passing over the bridge might crash. If this were a life saving medicine or procedure, one out of every 20 persons would die. Notice how large an error term is being allowed in the media research being shown on television (predicting election results or how many folks feel a certain way, etc.) Any error rate of more than plus or minus 1% is just a guess…. Most surveys you see advertised usually settle for plus or minus 3%… just not all that predictive….
Just keep things in perspective. What we are looking for/accepting in educational circles is not all that error free. This may be why much of the hyped new ‘findings’/methods in education are just that… hype….
Doing Diligence
Even though we are accepting such a high error rate, we still need to be doing due diligence when conducting/analyzing research findings. There are certain factors you need to look for
In research, data collection is integral to being able to draw any significant conclusions from the data. If it is not undertaken in a systematic way and with a clear purpose, it is not good/tight research. While data may be collected in many different ways and from a variety of sources that is assembled in a single document and the sources identified, it is still not considered ‘good’ research if the data are not interpreted. Thus, a good study has the following characteristics:
- Data is collected systematically.
- Data is interpreted systematically.
- There is a clear purpose
Two important aspects of research are that it is systematic and aims to find out things. Systematic implies that research must be undertaken using logical relationships and not just beliefs. Systematic and logical relationships involve an explanation of the methods used to collect data, giving the reasons and meaning of the results obtained and clearly explaining any limitations that affected the results.
Sam Messick and Lee Cronbach, two noted researchers who worked together often, believe in research there are only two principles: validity and reliability. Everything else follows… they have made a believer out of me… just think about it…
- Reliability is the converse of the definition of insanity… the expectation is that the results will be the IDENTICAL if you apply the same treatment over and over… (data is collected systematically).
- Validity means you are asking the right questions or attempting to solve the correct problem (there is a clear purpose)…
All the things you study in your research classes boil down to that… all the ways to disrupt validity (wrong group of folks being tested/treated… not represented… ignoring certain portions of the population…. No randomization… meaning if participants are not systematically included the results cannot be ‘generalized’ to any larger population… etc)
Or reliability (giving away the answers by not asking them in the correct way.. not asking questions repeatedly in different ways to be sure the questions were interpreted correctly, etc)
Minding the Gap
In academic research, usually a gap is identified in the literature and this gap reflects the lack of knowledge or limitations in the current body of knowledge within a discipline. As a result, the objective of conducting a research study is to generate new knowledge to fill that gap.
Types of research
1- Experiment – In this method, a researcher will manipulate an independent variable in order to determine whether it has an impact on a dependent variable. There are two types of experiments: laboratory experiment and field experiment. Laboratory experiment is conducted in a contrived situation while field experiment is conducted in a real-life situation. In any experiment, the factors are controlled (set constant) and only one factor is allowed to fluctuate so that its effect on the other factors could be seen. An experiment has very high internal validity.
2- Survey – A survey is a method which investigates the opinions and feelings of people. It involves interactions between the researcher and the environment. Information is collected through questionnaire or interview in many cases. Information collected in this way may not be available under any other circumstances. The relationship between the variables discovered through a survey research does not represent a causal relationship. The researcher can only claim that these variables are correlated. They cannot be certain that there is causation between the variables. Thus, the survey method has lower internal validity than an experiment. But there are ways to increase validity. One is to use the survey and add open ended questions at the end to cross check the results. This is often called multi-modal research.
3- Case study – In this method, a researcher makes a detailed study of a single case. The researcher usually aims to provide in-depth understanding of the specific features of the case and its related settings. Data are collected through observation, interview and document search. There are many concerns regarding the external validity of the case study method. One of them is – How could a single case be generalized or represent many more cases? However, many case-based researchers try to study a few cases in order to claim theoretical generalization. Other researchers argue that unlike the survey method, generalization is not the purpose of the study. In a survey, researchers generalize the findings to a larger population. In a case study, researchers seek to discover in-depth features of the case which cannot be acquired under any other circumstances.
So what is Action Research?
At FGCU some of you have taken or will take a course in action research. In this kind of study you are trying to apply generalized findings found in the literature to your specific (or yours and your colleagues’) circumstances. The converse is that it is Only going to be generalizable to your current activity. No claim is made that it is generalizable to a larger audience/population. In this regard it is very much like a case study. You are not looking at true experimental design (this supposedly has already been done).. you are in a way, APPLYING the research
Why do the research?
The purposes of doing research can be classified as follows:
1- Reporting / Exploring – This is the most basic or initial purpose of research; it is done mainly to collect initial or background information. In reporting/exploring the information, statistical summaries do not give inference or conclusion. Although some say that reporting is not a research, but if the systematic process has been followed then reporting studies are called research.
A research design does not need to be complex to have inferences .. in fact, often, the simpler the better…
2- Descriptive – The purpose of this research is to answer questions relating to who, what, when, where, and sometimes, how. Research is conducted to describe or define a subject, often by creating a profile of a group of problems, people or events. The study may involve data collection and creating a distribution of the number of times an event takes place or the characteristics of an object, or the interaction between two variables. This type of study may or may not have the potential to draw powerful inferences. If sufficient data is kept in the databases, the organization can easily conduct descriptive studies using internal information. Descriptive research is popular because it can be done across disciplines, and can be utilized easily for planning, monitoring and evaluating purposes.
3- Explanatory – The purpose of this research is to answer questions relating to why and how; it goes beyond description and explains the reasons for the phenomenon (a fact) that the descriptive study only observes.
4- Predictive – The purpose of this research is to predict when and in what situation something can happen. If the reasons for an occurrence can be explained, then it is also possible and desirable to predict when and in what situation the event will occur. Being able to predict the time and the situation of an occurrence makes it possible to control the phenomenon (a fact). The knowledge generated from the occurrence, called a phenomenon, can be used for other different groups of phenomena only if the researcher could take into consideration all other variables influencing the applications.
What to look for
Good research must follow a set of standards. The characteristics of good methodology are:
(a) Purpose is clearly defined to eliminate any ambiguity. The purpose may be to solve problems or to make a decision. State the purpose in writing; hence the statement to indicate the purpose should include its scope, limitations and precise specification of the meaning of all words and terms the researcher wants to use in the research. It is necessary to make sure the readers understand the research report and do not have any doubt as to what the researcher is doing.
(b) Research process is detailed so that any other researcher can repeat the research. The sources of data and the method of collecting the data should be described in detail, unless imposed by secrecy rules. If the process is not described in detail, then questions relating to the validity and reliability of the process and the data will arise.
(c) Research design is thoroughly planned so that the findings are as objective as possible. If primary data collection is to be used, the sample must be described to show evidence of its representativeness. If data collection methods lead to ambiguity, they should be avoided and replaced with available documentary sources or a direct observation method. Sources of information should be as thorough and complete as possible. When using experiments for data collection, the control observation must be satisfactorily placed. Efforts must be made to reduce personal influence or biases (validity and reliability issue) in selecting and recording data.
(d) High ethical standards are used to safeguard participants from any physical or mental harm. The design of the procedures and instruments should place high priority on privacy, confidentiality, dignity and no deception to ensure that no physical or psychological harm can happen.
(e) Limitations are frankly revealed and outline the affect of the constraints, or flaws in the procedure, design and data that might affect the findings. Some limitations may not have much significant effects on the findings, while others may cause the research to be invalid. If a researcher says that he does not face any limitation in doing the project, his research must be questioned.
(f) The analysis are adequate and appropriate to analyze the data and the analysis should be able to reveal adequate information to help the decision-making process. To ensure that the findings are sufficient, the validity and reliability of the data must be checked. The presentation of the findings should be restrained (i.e., not generalized more than the data reveals), clear, precise, comprehensive, and easily understood and organized.
(g) The conclusions are justified by the data of the research and limited to those that have factual basis. When sampling is used, caution must be taken not to simply make generalizations from the limited population and apply it universally.
Adding ‘power’ in research studies
This is where a good stats person comes into play. There are several things you can do statistically (or qualitatively) to add ’power’. For example, a small sample size (referred to as a small ‘N’) can be overcome with groupings of findings by common types. In a survey that has multiple ways of asking the results the same question the results of those questions that are repeated can be grouped. So, if you have an ‘N’=250 and have four questions all asking the same question, you, in essence, have similar power in the response as if you asked one question of 1,000 participants (I admit that I am over-simplifying here).. but you get my drift.. a good stats person is worth their weight in gold…
Ok, OK why are we reviewing this?
The reason for this class is that we will be looking at some studies that make some claims about human behavior and interactivity. The book the ‘Media Equation’ (the other lesson in this cycle) is making some claims here that are significant to our discussions. When I present their findings, I want you to decide for yourself whether what they claim is true (results are reliable and valid) based on their methodologies. I have my opinion… and others have theirs… I will present some of all of it… the process of interpretation that you go through may just prove the worthiness of including this stuff as a part of our class as the conclusions themselves.
What’s Next?
From here go to the lesson module on HCI in which further set the table for your review of the media equation premise.