Public Health Law Research Guide

The data's reliability and strength are important.  Some terms to know and things to consider:

• Opinion vs. scientific research
  Don’t assume research is scientific just because it follows some sort of methodology.  Scientific research is conducted systematically and follows the rules of inference.  Watch out for articles that cherry-pick data and sources, or that do not challenge their own viewpoint by looking at the support for the opposite side or alternate explanations.

• Replicability
  Data is considered reliable if someone else can replicate the results.  For this to happen, the researcher needs to:

  • Describe in detail the data set used,
  • Describe in detail how the data was measured, and
  • Minimize judgment calls – for example, set out detailed instructions for coders if part of the research involves classifying court opinions.

• Validity
  Does the data actually measure what is supposed to be measured?  If a data source is a proxy for what the researcher wants to measure (such as using political party affiliation as a proxy for a legislator’s personal political views), what does the researcher do to ensure it’s a valid, unbiased source of information?.

• Counterfactual studies
  A pure counterfactual study compares two identical experiences, where the only difference is the single condition being studied.   Because such a study is impossible to replicate, particularly in studying the effects of a law's implementation, legal researchers often end up comparing a population before and after a new law is applied to it.

• Experimental studies (aka random assignment)
  This is a study in which the researchers randomly assign people to different groups, and the experimental condition is applied only to one group.  The group to which the experimental condition is applied is called the “treatment group” and the group to which the condition is not applied is called the “control group.”

• Observational studies
  This is a study not using random assignment.  An observational study can strengthen its reliability through other experimental design measures such as:

  • Making multiple observations,
  • Comparing the law’s effect on different groups,
  • Comparing jurisdictions with and without the law,
  • Examining whether similar laws had the same or different effects when enacted in different jurisdictions, and/or
  • Comparing outcomes for targeted behavior and non-targeted behavior.

• Causation vs correlation accounted for and discussed
  Just because X and Y happen in tandem does not mean X caused Y.  Researchers need to explore whether there are other variables at play and test any supposed causal relationship.  (Is there a Z that caused both X and Y?  Does X cause Z, which in turn causes Y?  Could Y cause X?  Or is it possible that X and Y’s occurrence together is a coincidence?)