Now that we have all this data collected about media and aggregated in your profiles, the question now becomes how do we analyze it so it becomes useful information?
We break this module down into focus areas. The first is a general review of the processes and research principles that we use to analyze media and how to measure our responses to it and the possible effect they have on our brains. To do this we need to first cover a couple general analytical axioms related to sampling, validity, and reliability. We promise not to delve too deeply into the statistics behind research as those are well covered in other statistics courses you can take. But these three (which I will call the ‘essential trio’ for analytical design) are important because without these no research study is worth the time spent administering it. As my statistics professor once said, “in research, validity and reliability is everything, anything else is… (you fill in the rest)” Actually he was quoting his own professor and mentor, Dr. Robert Snow (of the Cronback and Snow fame).
Following the general introduction to formulating a research study, we move on to specifics about how we go about analyzing media and our responses to them.
Last, we will take a look at a couple ancillary views about media (especially visual media) in the form of looking at what some refer to as visual intelligence, the co-called ‘new brain’ and the interaction of media and cognition. I think these will be a great way to complete the study we are conducting this cycle.
The intent of this course is simple: if you can get a handle on the major characteristics of each media type and define them for yourself and how they interact with humans, you will have a sound basis for selecting the media type for your instruction. Using the checklists that based on your final notebook you can then move on to making better decisions regarding which app to use in your class and whether it has the best chance of assisting you in achieving your teaching goals.
To help you know what to read for in the sections that follow, you may first wish to look at the “Do This” box at the end of this lesson before begin reading.
We are surrounded by people making known their opinions about a subject. While this is protected by the Constitution under free speech, nothing gives education more of a bad name than when folks provide them when they have little or no basis in research. One of the problems with doing educational research is that we cannot conduct it as purely ‘experimental’ (i.e., with true treatment and control groups. For one, that is illegal in many states… you simply cannot provide a known treatment to one group that you strongly suspect is going to either have no effect or is harmful… so, the concept of ‘quasi-experimental’ was born. The concept of experimental being quasi or ‘almost is anathema to purists and may be the cause for many of them to consider educational research being a ‘soft science’.
While this may be true in many cases where folks take too many shortcuts in an attempt to get their dissertations of journal articles published, there are many things one can do to ensure the validity of their studies and are discussed in detail below.
But first we need to agree on some basic ideas about research in general.
How Not to Be Wrong
What this entails is the idea that you may be looking for too much significance in your study and/or or over-designing it. Believe it or not, simpler is better. In his book on the subject, Ellenberg argues that perhaps we are asking too much with our studies. This is true especially in educational domains. First of all, there is the error term (or the probability that we are wrong). In education, we are satisfied with a 5% error. We learn in our stats courses that there is a risk-reward factor embedded into what is an acceptable error term. What this means is that if you are doing some mid-level research the error term one can cope with can be higher. Think about it… if we were inventing an new medicine and were satisfied with a 5% error, that means that one out of 20 who take this new life-saving drug may die. Not good enough for medicine but may be ok in our field. So much for teaching humility! We are not deprecating the value of educational research but simply letting you know why validity is so important. Validity, reliability and sampling etc, is because we can control what we can control and that makes our work that much more valuable.
We have downloaded a pdf of the Ellenberg’s book knowing that you will most likely not read it all. The best parts are sections eleven, seventeen and eighteen as well as his intro in which he explains his premise.
Download Link: http://emeclasses.org/wp-content/uploads/2016/10/hownottobewrongsmall.pdf
Causation
As we have learned, a moving image is worth an infinite number of words. The following is offered to help you understand the concept without overloading you with more text in this lesson. The video plays for about 15 minutes. The Annenberg series (Against all Odds) is an award winning series.
As we said previously, at the heart of sound research are three main elements:
- Blocking and Sampling
- Validity
- Reliability
Doesn’t sound like much but these three are at the heart of any good research study and the first thing reviews and dissertation committees look at when evaluating the study. Surely the wrong analysis tool can be utilized which will skew the study. But noting can do more harm (or make the study more valuable) than careful attention to these three elements.
Blocking & Sampling
Annenberg Learning Center is also the place we go to find the best lesson on blocking and sampling. This video is quite old, but the concepts have not changed. I had saved it on my server due to its age and fear that id might disappear. I was not wrong. This one you will not find anywhere else. I offer it because it introduces the topic in the easiest way and is very understandable.This one is about 30 minutes but is worth it.
Validity and Reliability
These two are generally looked at together. YouTube has tons of videos on the subject. The hardest thing I had to do was to find one that is presented at the right tone and on the right level. I found one that lasts 39 minutes and is excellent but perhaps too scholarly for the time we have to look at this. There was one done by someone who is obviously a 'hip' millennial who does the whole thing in about four minutes... maybe too lighthearted for us. The one I provide is one of the better ones. This one lasts about seven minutes.
Validity is an important enough topic that also can be covered on its own. Again in digging for a video I came up with this one... there are others if it does not suit your taste.
Analyzing Text
First of all, we must note that as educators, we are accustomed to measuring/assessing the validity of our interventions through testing (whether in-house , end of course, or standard). We differentiate it here because, as instructional designers we do not have the luxury of waiting until a course is completed before we determine what we designed (in this case the media we select) is effective. Designers are faced with needing to find tools that will help us predict outcomes well in advance. And because we do not always have the time to test our design decisions, we must utilize what has been found to be effective in the literature. What follows, then is a series of readings that demonstrate what has been done and to give you means to be able to locate literature on your own and make determinations as to whether what is being provided has value (i.e., is valid and reliable).
In order to expedite that process we borrow considerable information from communications studies because they tend to provide the closest predictability that we are looking for. We also wish to make it clear that self-reporting tools such as those found on surveys are useful as a part of a mixed method studies but more and more experts in the field are deprecating their effectiveness as reliable validity tools. Surveys are very useful and probably the most often used in education studies. What to look for here are instruments that measure responses of a different kind... not responses to surveys but those measured innately as the user/reader/viewer/listener/player actually performs the reading/listening/viewing/gameplay.
One of the means to measuring/predicting user/reader responses to text is through an analysis of attention. Several products can be found on the market that measure attention. One could also utilize eye movements, heart rate etc. (as will be shown in Lang's book that follows). One of the more recent additions to the market is a product called Nervanix. We are not necessarily recommending Nervanix here but we introduce it because we have had first hand experience with the product and it appears to be useful and has potential. Our research with this product continues. We do know that is ==t is based on brain-based research and has tremendous potential. It is affordable and easily implemented.
Link to their website:
http://nervanix.com
Measuring Responses to (non-text) Media
We generalize as to the types of media that are discussed in this section as, for the most part, the kinds of studies that are organized can be the same or at least very similar. First, we discuss Annie Lang's work (she is actually the book's editor) on measuring media (mostly visual) but it is relevant to immersive media also in some cases. The goal is to demonstrate the kinds of studies that have been found in the literature to be the most effective. we are not asking that you become an expert in this arena, as the subject is normally taught as a part of an entire course or series of courses that form a degree in communication studies. Our job is to provide some ideas to help you organize your thoughts as you begin to select the media for your courses.
In this book, two parts provide the most insights: The Preface and the Chapter on setting up experiments.
Besides the passages that are provided below, the book's contribution to our current review at this is a look at the table of contents as described in the preface:
Download Link: https://emeclasses.org/wp-content/uploads/2016/10/measuring-media-preface.pdf
One of the most relevant chapters in the book is Chapter Nine. In it Byron Reeves and Seth Geiger discuss setting up various experiments to assess psychological responses.
Download Link: https://emeclasses.org/wp-content/uploads/2016/10/measuring-media-experiments.pdf
Analyzing Media
Cheseboro and Bertlesen's book provides additional information of how one can compare the usefulness of media messages by way of outlining the history of media as a communication tool and its effect on human culture. Their preface/introduction outlines a short but exceptional view. This view of measuring media speaks to the need to analyze the learner in your instructional designs. Analyzing each media type in terms their interaction with the user/viewer/reader/listener/learner/player-learner should be a good addition to your notebook.
Download Link: https://emeclasses.org/wp-content/uploads/2016/10/analyzing-media.pdf
A second portion of the book offers an analytical view of media through an overview of the theory of mediated communication.
Download Link: https://emeclasses.org/wp-content/uploads/2016/10/mediated-communication.pdf
Media Messages
Recognizing that we may be straying a bit, we can learn something from Riffe et al's book on Analyzing Media Messages. This book is a scholarly review of how to look at media messaging and provides a quantitative view on content research. While most of content analysis in communication studies has limited value for our purposes, the introduction does help to open you up to considering the intersection of the medium used and the message content, something that perhaps is worth adding to your notebook.
Download Link: http://emeclasses.org/wp-content/uploads/2016/10/analyzing_messages.pdf
When considering doing research in educational arena we are often faced with deciding which is the best methodology we can select? Knowing the limitations of being able to do purely experimental research, what is it we can actually do?
Two categories that are worth mentioning for our studies are action research and educational design research. Both are coming into their. Both follow the principles of formative design and research, which is very relevant to an instructional designer's role. With that in mind we should explore all three ideas.
Action Research
Download Link: https://emeclasses.org/wp-content/uploads/2016/10/action_research.pdf
Formative Design/Analysis
Educational Design Research
We have brushed upon the previous two topics so as to provide you with some insights about current trends in educational research. Educational Design research deserves a 'deeper dive". If we were to ask you to purchase one book for this course, this might be it.
Download Link: http://emeclasses.org/wp-content/uploads/2016/10/EducationalDesignResearch.pdf
The following authors bring to the table additional thoughts on how we analyze the impact of technology decisions we need to make as a part of our instructional designs. I will leave it up to you as to their impact on the final content you include in your media notebook. Please do not take them lightly. While much of the discussions follows a communications studies path, a lot of it correlates with our thinking on pedagogy. The issue of approaching/considering media and technology as ecosystems as a part of our design is a complicated one. Hopefully, we have been able to make that point with you during this semester.
- Gavriel Salomon - Interaction of Media Cognition, and Learning (Symbol systems studies)
- Richard Restak -the New Brain (Amazon Look inside)
In fairness we need to offer an opposing View
- Anne-Marie Seward Barry - Visual Intelligence (Amazon Look Inside) - Chapters 5 & 6 most relevant
- Brian Goldfarb - Visual Pedagogy (Chapter One Most Relevant)
- Richard Mayer - One last consideration for your notebooks ... what about combining media types?
Write your response and follow the instructions found in the drop box on canvas.
References
Barry, A. M. (1997). Visual Intelligence: Perception, image, and manipulation in visual communication. Albany, NY: State University of New York Press.
Chesebro, J. W. & Bertlsen, D. A. (1996). Analyzing Media: Communication technologies as symbolic and cognitive systems. New York: Guilford Press.
Ellenberg, J. (2014). How not to be wrong: The power of mathematical thinking. New York: Penguin Press.
Goldfarb, B. (2002). Visual Pedagogy: Media cultures in and beyond the classroom. Durham, NC: Duke University Press.
Lang, A. (ed.) (1994). Measuring psychological responses to media. Hillsdale, NJ: Lawrence Erlbaum Associates.
McKenney, S. & Reeves, T. C. (2012). Conducting educational design research. New York: Routledge.
Restak, R. (2003). The new brain: How the modern age is rewiring your mind. Emmaus, PA: Rodale.
Riffe, D., Lacy, S. & Fico, F. G. (1998). Analyzing media messages: Using quantitative content analysis in research. Mahwah, NJ: Lawrience Erllbaum Associates.
Salomon, G. (1994). Interaction of media, cognition, and learning: An exploration of how symbolic forms cultivate mental skills and affect knowledge acquisition.Hillsdale, NJ: Lawrence Erlbaum Associates.