Each student will be designing, conducting and writing up the results of a research project. The
objectives are both to apply some of the sociological knowledge learned during the course, and
to gain an appreciation for what is involved in the practice of sociology through the pursuit of an
actual research project. The particular type of research project students will pursue is a
quantitative newspaper content analysis. Content analysis is the study of recorded human
communication. This includes detailed, systematic analysis of “text” to identify patterns or
themes. Content analysis examine data in order to understand what they mean to people, what
they enable or prevent, and what the information conveyed by them does. A quantitative analysis
differs from a qualitative analysis because it attempts to ‘quantify’ the problem by way of
generating numerical data you can then use to uncover patterns in research. We do this by
classifying features, counting them, and attempting to explain what is being observed. The skills
and knowledge necessary will be developed through weekly tutorials taking place between
September and March, and digitally on Portal through your own TA’s Corner. Your portfolio
will contain your work-in-progress through the various stages of the project. Your ultimate
research report you will submit for assessment at the end of March will include sections on your
topic/literature review, on your designed research question and its rationale, on your
operationalization/coding scheme, on data collection/sampling, on your findings, on your
discussion of these findings and offering a tentative answer, on the strengths and weaknesses of
your research project. Completing this project you will learn about the key role played by each of
these aspects of the research design process, and how together can possibly lead to sociological
knowledge.
The research report HAS to be between 7.5 and 8 double-spaced pages (not including title page,
bibliography, and the appendix), 12pt font (Arial or Times New Roman), with one inch margin
all around. Note: turning in a longer report is not an option. Part of the difficulty associated with
the research report writing process is summarizing your research in a concise way by making
decisions about how to best use the space you have.
The outcome of your work needs to be written following the structure of a research report which
should include the following sections, in order (you will use these as sub-heading in your report)
and sticking to the page requirements:
Title page (does not count towards 7.5-8 pages)
1-Introduction (½ page)
2-Topic/Lit Review (1 page)
3-Research Question/Rationale (½ page)
4-Operationalization/Coding Schemes (2 pages)
5-Sampling/Data Gathering (½ page)
6-Findings/Crosstabs (1 page)
7-Discussion/Data Analysis/Answer (1 page)
8-Reflection/Strengths/Challenges (½ page)
9-Conclusion (½ page)
Bibliography (does not count towards 7.5-8 pages)
Appendix (does not count towards 7.5-8 pages)
Title Page (does not count towards 7.5-8 pages)
Title pages are pretty straight forward, and is the first page of your research report. While there
are different models out there, one will be circulated by your TA closer to the deadline. It will
include various pieces of information, namely your name, student number, an appropriate title
for your report, course code (SOC101Y), University of Toronto, the date, instructor’s name, your
TA’s name. Nothing too fancy, just this identifying information.
Introduction (½ page)
The introduction is half a page. Keep in mind you are not writing an essay, this is a research
report. The introduction is not a full of broad statements about the importance of Sociology or
the role of society in our lives. A research report’s introduction is short, to the point, and fairly
standard. In the introduction, you state what your research project is about. In this case, it’s a
quantitative content analysis of Canadian news sources (you could specify which if you focused
on only a few sources). You will state what your topic is, what is the research question you are
investigating, a line two on what the main finding of your work is, and the rest of the half page is
you offering an outline of what follows in the research report. The Introduction is there to frame
your work. It’s there to help set expectations for your reader about what they are about to read.
You should keep editing that half page until it’s tight and every sentence has been deemed
essential. Then you’ll have a strong, concise presentation of your work setting the rest of your
report. This is the “road map” of your paper – you’ll want to show us what you looked at, what
you found, and what will be covered in the rest of your report. Again, you’ll want to avoid any
broad, or sweeping statements (i.e. “The study of gender in sociology impacts every facet of our
lives”), and stick to the essential information noted above.
Topic/Lit Review (1 page)
The literature review is a short review about what has been covered in the academic research on
your topic. The literature review has a few different functions aside from reviewing what work
have been done on your chosen topic. It also can highlight what questions and issues are debated
and important in that field, draw out what insights and findings have been garnered about this
topic, and provide potential ideas about research questions, methodologies and potential
operationalization for your own research. So it acts as an inspiration, and helps set the context for
your own eventual contribution and where your project sits in this larger body of work. This
collection of articles will be done mostly in collaboration with your tutorial, where each student
will be asked to contribute articles to a larger review. Then, you will pull from the academic
articles that have been reviewed that you find most relevant to your research, and you will use
those pieces to paint the background and context for your own work. That involves discussing
briefly the broad sociological topic within which your project fits in, and then the more specific
aspect you are investigating. It also involves discussing how things are presented in the media,
and the importance and value of investigating those critically (i.e. in this case, conducting a
content analysis of news media). This section helps lay out why the topic and the research
question you are investigating matters, in doing so it acts as the foundation for your research
question. If you think of your research question as the ‘foreground’ of a painting or photo, then
your lit review section is the ‘background’ of this same painting or photo.
Research Question/Rationale (½ page)
Your research question is the foundation of your research project. A research question is a
question which you do NOT already have the answer to, and a question that is answerable
through careful research. A research question is a clear, focused, concise, complex question
around which you center your research. Your research question should be focused. Research
questions must be specific enough to be well covered in the space available. Your research
question should be complex. Research questions should not be answerable with a simple “yes” or
“no” or by easily-found facts. They should, instead, require both research and analysis on the
part of the writer. A good research question is one that is appropriate to the space (one 7.5-8
page research report, not a book), time (written for a single course, not for a 4 year PhD), method
(content analysis, not interview, fieldwork or experiments) and object of study (news sources) of
your research project. Give what you are analyzing, content of news sources, it is a good idea to
have your research question focus on media depictions, something that is incredibly appropriate
for content analysis (i.e. a good format is to think “How does the media depict…..”). It also has
to be a sociological research question, meaning you want a research question that allows you to
investigate a sociological matter and displays the sociological imagination. Your research
question needs to posit the variables you are intending on studying, and it needs to be
measurable. That means when composing your research question keep thinking about how you
will go about measuring the variables found in your question. If you cannot measure those
variables by doing a quantitative content analysis of news sources, you need to pick a different
research question. Finally, your research question cannot attempt to establish a causal
relationship. No causality*.
*On causality: When studying more than one variable, you can study the relationship between
them. If the researcher can establish an association between two or more variables, whereas the
presence, absence or changes in one variable is linked to the presence, absence or changes in
another variable, you can establish a correlation. Simply put, correlation implies a relationship
or connection between two or more variables. Correlation, however, does not imply
“causation”, or the idea that one variable caused the resulting effect. Causation requires
direction, and it requires isolating that variable X caused the changes in variable Y. While
students can certainly seek to investigate a correlation between two variables, they should NOT
seek to establish a cause and effect relationship. It’s important to remember that a correlation
between two variables does not necessarily imply that one caused the other. Causality is
something very difficult to establish solely through content analysis, and not suitable for this first
year research project. So your research question CANNOT attempt to study the impact of one
variable on another. It can, however, attempt to establish a relationship between two variables.
This means that you should be very aware of the language you use to describe your findings.
The write-up of this section is where you re-state the clear and specific research question you
propose to investigate (it is a good idea to italicize your research question so it sticks out), along
with offering a rationale indicating why this question is appropriate for this research project.
Rationale means explaining why it’s an important research question. Use the various criteria of
good research questions (bolded and explained above) and speak to each of those criteria.
Research questions are the foundations of research projects. The more you can communicate to
your reader that you understand what a good research question is, and that yours fits those
criteria, the better you are going to do.
Operationalization/Unit of Observation/Coding Scheme (2 pages)
This section is at the center of your research design efforts. Discuss what are your units of
observation and units of analysis. Then clearly lay out how you will measure your key variables
looking at these units of observations/analysis. This can be done by creating a detailed coding
scheme for each variable that makes it clear how each category is constructed.
Units of observations and analysis are which aspects of newspapers you will be observing and
analyzing. Keep in mind news are composed of various types of content including headlines,
stories, advertisements, listings, content promotion, comics, graphics, photos, etc. Sometimes
units of observations and units of analysis are different, such as if you are observing news
headlines (units of observations), but analyzing individual words within each headline (units of
analysis). For this project, your units of observation and units of analysis are the same, which are
full length news stories. You will want to discuss this in your research report.
Now that you have your unit of observations and analysis, you will now need to turn your
attention to operationalization, the single most important step of the research design process.
Operationalization is the key step that links your research question and your analysis.
Operationalization is how will you measure/observe/notice/identify the elements (i.e.
variables) in your research question? You need to assign values to your variables. What will
count as evidence? What will you be looking for? Be as precise as possible.
Operationalization is the process of defining your variables into measurable factors.
Operationalization is the link between your research question and your data collection. It is how
you will answer your question. Your operationalization of those variables is what you will
actually be looking for when you go through your news stories, it is the designing of your coding
scheme, of creating the categories through which you will classify your data. A researcher will
want to ensure consistency so that she is coding for exactly what she wants to code for. Think of
it as developing a set of rules that helps the researcher ensure that she is coding things
consistently throughout the process, and the same way every time. This also ensures that the
researcher is not engaging in selective observations, noticing only information that fits in the
researcher’s pre-conceived ideas of what they expected to find. Part of this includes being open
to finding inconsistent information that does not “fit” their expectations or theories. Good
operationalization will help the researcher distinguish between relevant and irrelevant
information. Particularly, it will help you, the researcher, decide which information should be
ignored because it does not speak to the research question one way or the other.
In your research report you will communicate your operationalization through the presentation of
your coding schemes (one for each of your two variables). Each coding scheme should contain
four elements:
1. The name of each category you have broken down your variable into
2. A clear definition of each category
3. Illustrative examples of content that would go in each category.
4. It also presents examples that might seem to go in a given category, but actually belongs
in a different category and explains why (a non-example)
The first step of operationalization is breaking up a variable into categories. Whole
books have
been devoted to the subject but here are a few guidelines for this research project:
1. Categories should be mutually exclusive. This means you need to make sure that they do not
overlap.
2. Make sure there are meaningful differences between categories. Fine distinctions between
categories cause confusion and make results unreliable. A good example would be asking people
to categorize stories as: not local, somewhat local, primarily local or completely local. You can
bet that coders will get confused between “somewhat” and “primarily” and would sit and wonder
what a “completely” local story would be.
3. Make sure there’s a category for everything. If there’s no category, coders will try to force
stories into inappropriate categories and distort results.
4. Only create separate categories for things you see often. Don’t create a new category
specifically for a unique instance that only appears once. The goal is to be able to classify items
90-95 percent of the time. There will always be 5-10 percent of items that are odd – create an
“other” category and use it cautiously.
These last two points may seem to be at odds. It’s a balancing act between having too few
categories and miscoding items and having too many and creating confusion. The best answer is
to test categories using a few items (i.e. news stories). By coding a few dozen examples it will
become clear very quickly if you have too many examples that don’t have a category or
categories without any examples in them at all. Given that your research question has two
variables, it is recommended not to use more than four separate categories for each variable
(or three categories + “other”).
Good Coding Scheme Construction Requires Practice
While you will initially develop coding schemes collectively with other members of your
tutorial, you will tweak these coding schemes once you start engaging in data gathering. It is
recommended to take 10-15 news stories and practice coding them to continue to tweak your
coding scheme. Go through each item and think about how they were categorized – this is
extremely helpful in making sure that you have laid out clear definitions. Once you feel like
you’ve got a good handle on what you’re going to measure, and what the possible answers for
that measurement are, test it. Ask someone else (or a few people) to read five or 10 news stories
of yours and code them. Everyone should come up with the same coding categories the vast
majority of the time. If they don’t, you may need to reduce the number of categories to eliminate
overlap or make the definitions clearer. If that doesn’t help, you may be measuring something
that’s just too unreliable. Note: Testing your coding scheme is an essential part of the research
process. It is part of what differentiates conducting a scientific investigation instead of an ad hoc
one.
Good operationalization is when it is in line with your question and tells exactly you are going to
count/measure/observe. Your TA will help you make sure there is alignment between your
operationalization and your research question. That is a key factor in determining the quality of
your research project, whether this alignment is present, and to which extent. Make sure what
you are measuring/identifying/observing will allow you to speak to your research question. Once
you have your research question operationalized,
it is time to move to sampling and data
gathering.
Data Gathering/Sampling (½ page)
The next stage in the research process is data gathering (and sampling). Your quantitative
newspaper analysis must come from the following fifteen news sources: Globe and Mail,
National Post, Toronto Star, Toronto Sun, The Vancouver Sun, The Province (Vancouver),
Calgary Herald, Winnipeg Free Press, Ottawa Citizen, The Gazette (Montreal), The Hamilton
Spectator, Edmonton Journal, The Chronicle Herald (Halifax), cbc.ca, and ctvnews.ca, from
September 2016 until March 2017. You cannot possibly analyze all the sections from all the
issues of the fifteen news sources over a six to seven month period. Each student need to analyze
a sample of all the available material. Sampling is about selecting a portion (a sub-section) of the
whole population. Sampling is necessary only when analyzing the entire population is unfeasible
or unreasonable. It is selecting items that will tell us something more general than the specifics
items we’ve actually selected. Sampling fits under two broad categories, probability and
nonprobability sampling. Probability sampling is when a subset of cases/items is chosen at
random from a larger population. Also for it to be probability sampling all items in the
population have an equal chance of selection, there is then an excellent chance that the sample so
selected will closely represent the population of all elements. Probability sampling avoids
researchers’ conscious or unconscious biases in item selection. Under these conditions, results of
studies using this probability sampling can then be generalized to the whole population.
However, for this first year research project, due to the time and difficulties associated with
probability sampling, you should use nonprobability sampling. You are expected to find no less
than 50 separate articles which you will print out, analyze, and eventually include in your
research report (Note that you will need to include a copy of each article in your final research
report). Results stemming from nonprobability sampling, such as convenience sampling, where
researchers choose which subset of cases/items will be studied based on other criteria, cannot
scientifically be generalized to the entire population. Nonprobability sampling is used primarily
for reasons of convenience, cost, access, time or because the entire population of cases cannot be
easily identified. And while you are using nonprobability sampling, you must still lay out
explicitly in your research report how the sampling process was conducted, ie how you went
about making choices about how to select articles.
This section of the research report will be a description of your data gathering process, including
sampling. You will answer questions such as how many pieces did you gather, how you choose
them, and describe the timeline over which this took place. (Note: keep track of exactly where
you found each article, since you will need this information for your research report. It will save
you tremendous amount of time, if you keep good record of where everything is from as you do
it, rather than having to go back and re-find each piece later. Kristie’s Workshops will also have
advice on this very topic).
Findings/Crosstab (1 page)
Once you have gathered your data, you will code it using the coding schemes you designed
during the operationalization phase of the research process, and you will present your findings in
a crosstab table. Crosstab tables are used when a researcher is studying the relationship between
two separate variables, as you are doing in
this research project. When a researcher is
investigating a single variable on its own, he or she would be using a frequency table instead
(such as the table 1 just below).
Here is an example where the variable “Geographic focus” was coded based on categories
“local”, “state & region”, “national”, “international”, and “none”.
TABLE 1 example: Single Frequency Table based on One Variable (Geographic focus)
Geographic focus Frequency Percent
local 211 35.5
state & region 71 12.0
national 195 32.9
international 58 9.8
none 58 9.9
Total 596 100.0
In this table, we see that about 36% of the newspaper’s stories have a local focus, 12% have
state/regional focus, about 33% have a national focus, about 10% have an international
focus, and about 10% have no geographic focus at all.
However, given that your research question for this project is investigating the relationship
between two variables, you will need a crosstab (such as table 2 below). This is where you are
coding your 50 news articles using the two coding schemes you developed during the
operationalization stage of the research process. For example, if your research question was
investigating the relationship between the geographic focus of news stories with the source/type
of author of those stories, you would display this information in a crosstab. This will give you
the count and percent of news stories for each cell of the two variables, geographic focus and
story origin. (Note that it is also possible for research questions to have more than two variables,
but this process becomes far more complex to analyze which is why it is beyond the scope of this
course. All research questions for this SOC101 course are exactly two variables).
TABLE 2 example: Crosstab
Geographic focus and Story source crosstabulation
Geofocus Source of story
wire staff reader unknown Total
local Count 11 164 9 27 211
% within geofocus 5.21 77.73 4.27 12.80 100.00
% within Source of story 4.60 58.78 31.03 52.94 35.28
state&region Count 22 39 6 4 71
% within geofocus 30.99 54.93 8.45 5.63 100.00
% within Source of story 9.21 13.98 20.69 7.84 11.87
national Count 139 41 8 9 197
% within geofocus 70.56 20.81 4.06 4.57 100.00
% within Source of story 58.16 14.70 27.59 17.65 32.94
international Count 38 17 3 1 59
% within geofocus 64.41 28.81 5.08 1.69 100.00
% within Source of story 15.90 6.09 10.34 1.96 9.87
none Count 29 18 3 10 60
% within geofocus 48.33 30.00 5.00 16.67 100.00
% within Source of story 12.13 6.45 10.34 19.61 10.03
Total Count 239 279 29 51 598
% within geofocus 39.97 46.66 4.85 8.53 100.00
% within Source of story 100.00 100.00 100.00 100.00 100.00
Frequency tables and crosstabs allows you to see how many cases (such as news stories for
example) you have in each category for each variable. The purpose of this 1 page section in your
research report is to display your findings in the form of a crosstab, with the raw count and %
calculated for every cell, and describe your table using complete sentences. Any analysis or
attempts to make sense of the numbers should be reserved for the following section. (Note that a
separate document called Crosstab Walk Through will assist students in calculating the % for
each cell of your crosstab, as well as your tutorials.)
Discussion/Data Analysis/Answer (1 page)
This 1 page section of the research report is where you discuss the patterns and variations you
are observing in your crosstab. You will begin this section by re-stating clearly what is your
research question, and then use this section to make sense of the results in your crosstab in light
of your actual research question. Discuss what answer(s) they point to in regards to your research
question. Is it a straight-forward answer or are your findings pointing to a nuanced answer? Were
any of the relevant data you collected inconsistent with this answer? Note that since you will not
be running the statistical calculations necessary to determine whether the patterns/variations you
found are statistically significant, your discussion of your findings remain tentative, and not
definitive. This is very important to remember and to point out in your research report itself.
Data analysis is NOT a description of your coded observations – it is making sense of what you
found – interpretation is involved in an analysis not just description. Be sure not to overstate
your findings – findings should be data-driven, not vice versa. In the table 2 exemplar from
above, some of the patterns are that stories with a national geographic focus rely most heavily on
wire services, that most staff stories have a local geographic focus, that reader-generated stories
vary in their geographic focus, and that international stories also rely heavily on wire services.
Again, however, one would need to conduct a statistical test to see whether the differences in the
cells of the table are significant. This test and other statistical calculations are beyond the scope
of a first year course in Sociology.
Successful sections will be organized well, discussing the patterns and variations most relevant
to your research question. Remember this is not simply about describing what you see, you need
to “make sense” of what you see in light of your research question. This is the difference
between description and analysis. Data analysis is ultimately about recognizing the central
themes of your findings and making sense of them.
Your job as the researcher is to spend time
analyzing the evidence you have collected to try to answer the question you posed. And you need
to communicate this effectively, provide evidence that supports your tentative answer (again
definitive answers would require statistical calculations beyond this course) that can be evaluated
by your reader. In order for the reader to evaluate your analysis and answer, however, you need
to cite evidence to support it so that your reader can decide for themselves if your argument is
convincing.
Dont forget to talk about outliers in your data, the “this was not always the case” type of points.
“Although a majority of my data shows…there were still examples of…” This allows for the
complexities and nuances of your data to surface. Simple analyses miss out on the richness of the
data available. This is the section where a clear operationalization, thought out coding scheme,
and well collected data pay off with an interesting and nuance answer to your research question.
In almost all exceptional research projects for this course, the answer to the research question is
nuanced. You are highlighting not only overall trends, but smaller patterns, and findings that do
not “fit”.
Reflection/Strengths/Challenges (½ page)
This section is an opportunity to reflect on the months spent on this research project. What do
you believe are the strengths of this research project you conducted and what are some of the
challenges and obstacles you faced, and how did you navigate those or overcome attempted to
overcome them? You can discuss any steps in the research project, but use this section to
highlight for your reader what you might do differently if you knew when you started what you
know now. Part of the goal of this section is for you to reflect on the research process, and what
you have learned as a researcher conducting this quantitative content analysis. This section
should not just be focused on how you need better time management skills, but rather, should
reflect a challenging piece of the project and how you worked through it.
Conclusion (½ page)
This section is where you summarize your research project and offer some concluding thoughts.
You need to re-state your research question and summarize the tentative answer you have
obtained through your actual research project. All exceptional research reports will discuss how
your research project “fits” in the literature (academic articles you highlighted in your literature
review as most relevant and offering context to your project) and what contribution(s) your
research project make to this literature once its findings would be verified (using more advanced
statistical calculations). Finally, you should end your research report with a short note about what
you would do next if you were to pursue this research agenda further. What directions would
your work take to supplement to your current research findings. Note that you should never
introduce new information in the Conclusion section.
Bibliography (does not count towards 7.5-8 pages)
Referencing is about acknowledging all the sources which inspired or guided your work, and
about situating yourself as a researcher/writer within a broader intellectual, academic, political,
artistic, cultural field, etc. Your bibliography will contain every single academic article (used in
your lit review section or other sections), every single news articles you analyzed (all 50 of
them), and any other sources (textbooks, outside sources, including online sources, etc.) you used
in the development of your research project
or writing of your research report. The style to use is
the APA, which is the subject of one of Kristie’s Workshop you can find on Portal. It will guide
you through what information you need to display in your bibliography references, and how to
display them. The goal of any referencing style is to offer all the information necessary for a
reader to be able to locate the original if they’re interesting in reading it for themselves, or want
to find more information about an aspect of your project.
Appendix (does not count towards 7.5-8 pages)
Your appendix will include a copy of each of your fifty news article. For each article, you will
include a short-written caption (can be typed or handwritten) walking the reader through how
this article was coded and why. We do not require a novel, but enough information to know in
which cell of your crosstab was this article coded, and offer a rationale as to why this is the case.
You can use different colors, you can highlight relevant portion of your news article that led you
to make particular choices, but overall you need to be able to communicate effectively how each
your news articles were coded using your coding scheme categories and descriptions. You need
to demonstrate that you understand the coding process, how you went from operationalization to
produce the results in your crosstab. The better you can demonstrate this thoroughly, the better
you will do.
Submission:
Your quantitative newspaper content analysis write-up in the form of a research report is due on
March 27
th
. Three copies of the assignment must be submitted: 1) a hard copy to Room 225, 725
Spadina Ave, no later than Monday, March 27
th, at 3pm, 2) a digital copy through Portal tutorial
drop box (each tutorial will have its own digital drop box on Portal), and 3) Turnitin.com
(instructions for Turnitin this will be released in February). Note: There are no late penalties
per day late because research reports will not be accepted past the due date without
submission of UofT’s Verification of Illness form. Without this documentation, any
assignments submitted after the deadline will receive a grade of zero.
Resources
Note that these instructions are meant to be a foundation to assist in the conducting of your
research project and in the writing of your research report. However, it is not meant as a standalone
document. It is meant to be supplemented with advice you receive from your TA every
week in tutorial and on your TA Corner, as well as workshops posted by Kristie on Portal, the
weekly feedback document produced every Friday by Jenna, and other workshopping done
throughout the entire academic year. Tutorials will walk you through each step of the project,
and will be an important part to the completion of this project.