Social research methods: choose a design and audit a measurement
A research question is not yet a design, and agreement is not yet truth. Build a question-to-evidence plan, then examine how two coders’ classifications support a limited measurement claim. No real participants or private data are involved.
Course topic: Research Methods
Study route and exercise: University Reading List editorial project. How we curate · Sources checked:
Scope and responsibility
A first structured methods route for readers able to follow a social-science argument and elementary proportions. Bhattacherjee is a free graduate primer, Creswell/Creswell a sixth-edition proposal guide, and Salganik a first-edition digital-research companion. They do different jobs. The invented coding cases teach a defined nominal-agreement statistic; no fixed threshold is offered as a universal quality standard.
Choose your starting point
I need a free methods foundation
Use Bhattacherjee for the chain from constructs and theory to design, measurement, sampling and analysis. Supplement technique-specific detail when a project requires it.
I need to turn a question into a proposal
Use Research Design when comparing qualitative, quantitative and mixed approaches and writing a coherent plan are the immediate tasks.
My question involves digital traces or platforms
Use Bit by Bit after the basic design question is clear. Its historical examples are not instructions for current API access or permission to collect people’s data.
Compare the three books
Social Science Research: Principles, Methods, and Practices
Anol Bhattacherjee · Edition: 2 · 2012
Free compact methods spine
- Choose it when
- You need a connected map of concepts, measurement, design and inference.
- Use another route when
- You need a full software/statistics course or specialist qualitative analysis from one short primer.
- Selected reading
- Second edition: the theorizing, design, measurement, reliability/validity and sampling sections; locate the actual headings in the official text.
- Limits and trade-offs
- USF identifies a2012 graduate-oriented text and a noncommercial share-alike licence. “Beginner” here means beginning research methods, not no academic reading or statistics preparation.
Try before choosing
Choose an invented construct such as “response offers a reason”; write a coding rule and two borderline examples.
Check your work: The observable category is distinct from the construct. Explain where coder judgement enters and what a repeated classification could fail to establish.
Research Design: Qualitative, Quantitative, and Mixed Methods Approaches
John W. Creswell; J. David Creswell · Edition: 6 · 2023
Proposal and approach comparison
- Choose it when
- You need a research question, purpose statement and method to fit together.
- Use another route when
- You expect brief methods chapters to replace full technique training or assume mixed methods means merely having two datasets.
- Selected reading
- Sixth edition: 1,4 and 6–10 for approaches, ethics, purpose, questions and the three method families.
- Limits and trade-offs
- The publisher verifies the ten-chapter structure. It is a design/proposal book; a polished proposal alone does not show a measure is reliable or valid.
Try before choosing
Draft a purpose statement and then one design for a coding question; state which conclusion that design can and cannot support.
Check your work: The sampling unit, classification rule and inference match the question. A causal claim needs more than a two-coder agreement table.
Bit by Bit: Social Research in the Digital Age
Matthew J. Salganik · Edition: 1 · 2017
Free digital-research comparison
- Choose it when
- You want to distinguish observing, asking, experimenting and collaborating in digital settings.
- Use another route when
- You need current platform instructions or a first glossary before choosing a research approach.
- Selected reading
- First edition: 1.5 for the map,2–5 for contrasting designs, and 6 for ethics. Use the first-edition route rather than the older Open Review version.
- Limits and trade-offs
- The author emphasizes enduring research abstractions. Historical platform examples do not establish present access, consent or reuse rights; some activities need additional coding or mathematics.
Try before choosing
Use the author’s four-design map to propose two ways to investigate one invented question, and an ethical issue each creates.
Check your work: The designs change the researcher-participant relationship and the type of inference. Public availability alone does not settle whether collecting or combining records is appropriate.
Check your preparation
What counts as a unit?
Specify whether the unit is a person, post, paragraph, event or organization; changing the unit changes the denominator and often the claim.
Does agreement prove validity?
No. Two coders can agree on the same poor rule. Reliability concerns consistency; validity concerns whether the measure supports its intended interpretation.
What mathematics is used below?
A two-by-two count table, proportions and κ=(po−pe)/(1−pe). Here pe is the product-marginal baseline, not a demonstrated psychological mechanism of random guessing.
Six steps with a concrete output
Outputs marked ready: 0 / 6
These are your own study marks, not an assessment. They are not sent to this site.
-
Define the question and construct
Selected reading: Bhattacherjee’s theory/measurement sections.
Make this: Write a construct, one operational rule and a deliberately ambiguous invented paragraph.
Check your work: A reader can say how the rule assigns a category and where it leaves judgement.
-
Choose an approach
Selected reading: Research Design 1 and 6–10; Bit by Bit 1.5.
Make this: Compare two possible designs for the same question with their units, outputs and limits.
Check your work: The design follows the inference sought, not merely a preferred software package.
-
Make a small codebook
Selected reading: Measurement and reliability discussions.
Make this: Write mutually exclusive, exhaustive A/B rules for the supplied fictional items; keep a disagreement log.
Check your work: Rules can be applied independently before discussion; reconciliation is not presented as independent initial agreement.
-
Audit the first table
Selected reading: The original worked example below.
Make this: Compute margins, observed agreement, baseline agreement and κ.
Check your work: Every proportion has a stated denominator; no unprovided true label is invented.
-
Test the apparent improvement
Selected reading: The follow-up and the distinction between descriptive statistics and evidence.
Make this: Compare a new table’s raw agreement and κ with the first one.
Check your work: Changing item composition/margins prevents a simple claim of coder improvement.
-
Write a limited methods memo
Selected reading: Bit by Bit 6 and your proposal.
Make this: State sampling, coding, agreement and validity limits; list the next evidence needed before real research.
Check your work: No fictional example is reported as a participant observation, and no actual recruitment or scraping is performed.
Worked exercise
Two coders and 80 invented paragraphs
Both coders independently classify the same 80 invented paragraphs as A or B. Counts are A/A 36, A/B 4, B/A 12, B/B 28, where the first label is coder 1. Compute raw agreement and the stipulated unweighted Cohen κ, then state what is not established.
Assumptions
There are no missing classifications, and A/B are nominal, exclusive and exhaustive. The table is constructed, not collected. Define po as diagonal proportion and pe as the sum of products of the two coders’ category proportions. No true category or random sample is supplied.
- Observed agreement is(36+28)/80=0.80. Coder 1 assigns A to 40/80 and B to 40/80; coder 2 assigns A to 48/80 and B to 32/80.
- pe=(40/80)(48/80)+(40/80)(32/80)=0.50. Thus κ=(0.80−0.50)/(1−0.50)=0.60.
- There are 16 disagreements: 4 in one direction and 12 in the other. Without a reference truth or evidence about the codebook, this does not identify which coder is correct or whether the intended construct is measured.
Answer and interpretation
Raw agreement is 80% and κ=0.60 under the specified baseline. These are descriptive agreement summaries of this invented set, not accuracy, construct validity or a universal pass/fail rating.
Test the result
Recover both margins and 80 from the four cells. Changing to a reconciled consensus table would change the question; it must not be called the initial independent agreement.
Change an assumption
Higher raw agreement can accompany lower κ
Use a separate invented batch of 80 paragraphs with counts A/A 0, A/B 4, B/A 4, B/B 72. Retain independent classification, the same nominal categories and the same κ definition. No claim of a before/after intervention is supplied.
- Raw agreement is 72/80=0.90, larger than the first table’s 0.80. Each coder assigns A to 4/80=0.05 and B to 76/80=0.95.
- pe=0.05²+0.95²=0.905. Therefore κ=(0.900−0.905)/(1−0.905)=−1/19≈−0.05263.
- Most items are jointly labelled B, but there is no joint A. The margins and item composition changed, so raw agreement and chance-adjusted agreement need not rank the tables identically.
Answer and interpretation
The second table has 90% raw agreement and negative κ under this definition. Neither value alone demonstrates a better coder or valid measure; a label-prevalence shift can change the comparison.
Test the result
Check 0+4+4+72=80 and 0.905>0.900. Do not use a generic κ adjective as the conclusion; state the table, baseline and inference. No health, personality or social-group classification is performed.
Common mistakes
Treating a construct as its indicator
The label is an operational choice that requires justification, not the concept itself.
Making a percentage into truth
Agreement, accuracy and validity answer different questions; name the evidence each would need.
Treating digital access as permission
A technically accessible source does not settle research ethics, consent or contextual expectations.
Before moving on
- I can connect a research question, construct, unit and proposed observation.
- I can reproduce both coding summaries and explain why they are not validity tests.
- I can choose a main text by the task I need next and state one methodological training gap.
Sources and edition checks
Sources checked:
- USF: Social Science Research record and licence
USF verifies the 2012 free text’s graduate audience and CC BY-NC-SA3.0. The second-edition identity is retained; the route uses named sections, not a newly invented chapter map.
- SAGE: Research Design sixth-edition contents
SAGE’s sixth-edition contents verify chapters 1–10 and the proposal-oriented comparison of approaches. No full-book review or method effectiveness is claimed.
- Salganik: first-edition outline
The author’s first-edition outline and ethics chapter distinguish the four research approaches and their responsibilities. The 2017 publication/2018 copyright distinction is retained.
- Penn State: nominal agreement and kappa definitions
Penn State’s university teaching material gives nominal-agreement definitions. The original Cohen 1960 paper is identified in the research note; the two tables and all calculations are original and no clinical example is reused.
These are proposed study activities, not official course assignments or a claim that every book has been read cover to cover.