Quantitative Research Design
Session at a Glance
Survey design and questionnaire construction; probability and non-probability sampling; experimental, quasi-experimental, and correlational designs; secondary data analysis; validity and reliability
Developing and piloting a survey questionnaire; sampling plan design; secondary dataset exploration and analysis
2 hrs Lecture + 12 hrs Lab/Project
Draft quantitative instrument & sampling plan
Learning Objectives
By the end of this session, you will be able to:
- Select an appropriate quantitative research design — survey, experimental, quasi-experimental, correlational, or secondary data analysis — based on your research questions, paradigm, and feasibility constraints
- Construct a valid and reliable questionnaire using established scale development principles, including item wording, response formats, question sequencing, and pretesting protocols
- Design a rigorous sampling strategy by choosing between probability and non-probability approaches, calculating minimum sample size, and justifying your choices with reference to your population and analytical method
- Evaluate the threats to internal, external, construct, and statistical conclusion validity in a quantitative research design, and implement appropriate mitigation strategies
- Locate, assess, and prepare secondary datasets for quantitative analysis, including evaluating data quality, handling missing values, and documenting data provenance
Session Planner
Suggested breakdown of the 4-hour contact session.
| Time | Segment | Activity | Mode |
|---|---|---|---|
| 0:00–0:10 | Opening | Transition from proposal approval to methodology execution; overview of Unit 2 (Weeks 9–12); how quantitative designs fit into the methodology landscape | Whole class |
| 0:10–0:35 | Lecture 1 | Survey design and questionnaire construction — item development, response scales, question wording pitfalls, sequencing, pretesting; BBA and BCA examples | Lecture |
| 0:35–1:00 | Lecture 2 | Sampling strategies — probability vs. non-probability; simple random, stratified, cluster, systematic, purposive, quota, snowball; sample size calculation; sampling error and bias | Lecture |
| 1:00–1:25 | Activity | Questionnaire critique: given a poorly designed survey, identify and correct item-level problems (leading questions, double-barrelled items, ambiguous response scales) | Pairs |
| 1:25–1:40 | Break | — | — |
| 1:40–2:05 | Lecture 3 | Experimental and quasi-experimental designs; correlational designs; secondary data analysis; validity framework (internal, external, construct, statistical conclusion) | Lecture |
| 2:05–2:20 | Discussion | Selecting the right design — given 4 research scenarios, determine which quantitative design is most appropriate and justify | Whole class |
| 2:20–3:50 | Lab Work | Part A: Draft survey questionnaire or data extraction protocol; Part B: Design sampling plan with sample size calculation; Part C: Locate and evaluate a relevant secondary dataset | Individual/Pairs |
| 3:50–4:00 | Exit Ticket | Submit draft instrument and sampling plan; identify the most significant threat to validity in your design | Individual |
1. The Quantitative Research Landscape
Quantitative research is the systematic empirical investigation of observable phenomena via statistical, mathematical, or computational techniques. Its defining characteristic is not the use of numbers — it is the commitment to measurement as the basis for claims about relationships, differences, and patterns. If your RQs ask "how much," "how many," "what is the relationship between," or "what is the effect of," you are in quantitative territory.
A quantitative research design is the blueprint for collecting, measuring, and analysing numerical data to answer research questions or test hypotheses. It specifies: (a) the type of design (survey, experimental, correlational, etc.), (b) the variables and how they are operationalised and measured, (c) the population, sampling strategy, and sample size, (d) the data collection instruments and procedures, (e) the statistical techniques that will be used to analyse the data, and (f) the strategies for ensuring validity and reliability. A well-specified design enables another researcher to replicate your study — this is the gold standard of quantitative rigour.
1.1 Choosing a Quantitative Design — Decision Framework
| If Your RQ Asks... | And You Can... | Consider This Design | Example (BBA) | Example (BCA) |
|---|---|---|---|---|
| "What is the relationship between X and Y?" or "To what extent do A, B, and C predict Z?" | Measure variables as they naturally occur (no manipulation) | Correlational / Cross-Sectional Survey | Relationship between ESG disclosure scores and firm profitability (Tobin's Q) among Nifty 500 firms | Correlation between code complexity metrics (cyclomatic complexity, LOC) and defect density in open-source Python projects |
| "What is the effect of X on Y?" or "Does treatment A produce better outcomes than treatment B?" | Manipulate the independent variable and randomly assign participants to conditions | True Experiment (RCT) | Effect of discount framing (% off vs. absolute amount off) on purchase intention, randomly assigning 200 shoppers to one of two conditions | Effect of code review tool (manual vs. AI-assisted) on bug detection rate, randomly assigning 30 developers to conditions |
| "What is the effect of X on Y?" | Manipulate the IV but CANNOT randomly assign (pre-existing groups, natural intervention) | Quasi-Experiment | Effect of a mandatory financial literacy course on savings behaviour, comparing students who took the course (treatment) with those in a different programme who did not (control) | Effect of migrating from monolithic to microservices architecture on deployment frequency, comparing teams that migrated (treatment) with teams still on monolith (control) |
| "What are the characteristics, attitudes, or behaviours of population P?" or "How prevalent is X in group G?" | Collect data from a sample of the population at one point in time | Descriptive Survey | Prevalence and patterns of digital payment adoption among urban vs. rural consumers in Karnataka | Adoption rates of specific programming languages and frameworks among Indian software startups |
| "How does X change over time?" or "What is the trend in Y across periods?" | Collect data from the same entities at multiple time points, or access longitudinal data | Longitudinal / Panel Study | Change in consumer trust toward e-commerce platforms before, during, and after the pandemic using repeated cross-sectional data | Evolution of technical debt in a software project across 12 consecutive release cycles |
| "What does existing data reveal about RQ?" | Access a high-quality secondary dataset that contains relevant variables | Secondary Data Analysis | Analysing NSSO consumption expenditure data to study changing spending patterns; using ProwessIQ for financial ratios | Analysing Stack Overflow developer survey data to study technology adoption trends; using GitHub Archive for open-source contribution patterns |
The most common quantitative design error is choosing a method first and then trying to fit research questions to it. "I want to do a survey" is not a research question — it is a method preference. Start with your RQs. Ask: "What kind of data do I need to answer these questions? What design can produce that data?" The design is a tool, not a destination. A student who can articulate why their RQs required a particular design demonstrates methodological competence. A student who has a design but cannot explain why it fits their RQs does not.
2. Survey Design & Questionnaire Construction
The survey is the most widely used quantitative method in BBA capstones — and among the most poorly executed. A well-designed survey is a measurement instrument, no different in principle from a thermometer or a voltmeter. A poorly designed survey produces numbers that look like data but are actually noise.
2.1 The Questionnaire Development Process
Before writing a single question, define what you are measuring. Each construct in your conceptual framework must be: (a) defined conceptually (what does it mean?), (b) operationalised (how will you measure it?), and (c) broken into dimensions if it is multi-faceted. "Customer satisfaction" may have dimensions of product quality, service quality, and price perception. Each dimension needs its own items. Output: A construct map showing each construct, its dimensions, and the number of items planned for each.
Adopt first, adapt second, create last. For established constructs (job satisfaction, perceived usefulness, trust), use validated scales from published research. These scales have been tested for reliability and validity — your self-created items have not. Cite the source of adopted scales. If you must create items (for novel constructs or context-specific elements), generate at least twice as many candidate items as you plan to use — half will be eliminated during pretesting. Output: An item pool with source citations for adopted items.
Likert scales (1–5 or 1–7 agreement/frequency) are the most common, but not the only option. Semantic differential scales (opposing adjectives), multiple-choice, ranking, and open-ended numeric responses each have appropriate uses. Key decisions: odd vs. even number of points (odd allows neutral; even forces a lean); fully labelled vs. endpoints-only; number of points (5-point is standard; 7-point captures more variance but increases cognitive load). Output: A consistent response format decision with justification.
Administer the draft questionnaire to 5–10 people similar to your target population (not friends who will be polite). For each item, ask: "What did you think this question was asking?" If their interpretation differs from your intention, revise the item. Time the completion. Identify items that cause hesitation, confusion, or irritation. After pretesting, conduct a reliability analysis (Cronbach's alpha) on pilot data if your sample is large enough (n ≥ 30). Output: A refined questionnaire with evidence that items are understood as intended.
2.2 Item Writing — The Rules
| # | Rule | Bad Example | Good Example |
|---|---|---|---|
| 1 | Use simple, concrete language — write for the lowest likely reading level in your sample | "To what extent do you concur that the omnichannel integration facilitates a seamless and frictionless consumer journey?" | "How easy is it to switch between the company's website, app, and physical store when shopping?" |
| 2 | Avoid double-barrelled questions — each item should ask about ONE thing | "The website is easy to navigate and visually appealing." | Split into two items: (a) "The website is easy to navigate." (b) "The website is visually appealing." |
| 3 | Avoid leading questions — don't signal the "correct" or desired answer | "Most experts agree that regular exercise improves mental health. How often do you exercise?" | "How many times per week do you engage in moderate or vigorous physical activity for at least 30 minutes?" |
| 4 | Avoid double negatives — they confuse respondents and produce unreliable data | "I do not disagree that the company should not reduce its CSR spending." | "The company should maintain or increase its CSR spending." |
| 5 | Ensure mutually exclusive and exhaustive response options | Age: 18–25, 25–35, 35–45, 45+ (overlaps at boundaries) | Age: 18–24, 25–34, 35–44, 45–54, 55 or older |
| 6 | Avoid assumed knowledge or behaviour — include "not applicable" options | "How satisfied are you with your manager's feedback?" (assumes the respondent has a manager and receives feedback) | "Do you have a direct manager who provides performance feedback? [Yes/No]." If Yes: "How satisfied are you with the feedback you receive?" |
| 7 | Keep items short — aim for ≤ 20 words per item | "Reflecting on your experiences over the past twelve months across all channels through which you have interacted with the organisation..." | "In the past 12 months, how satisfied have you been with this company's customer service?" |
| 8 | Vary positively and negatively worded items (but carefully) | All items worded in the same direction creates acquiescence bias (respondents agreeing without reading) | Mix: "I find the app easy to use" (+ve) with "I often get confused when navigating the app" (−ve). Note: reverse-coded items require careful handling and can confuse respondents if not clearly written. |
2.3 Questionnaire Structure — The Flow
| Section | Purpose | Content | Typical Length |
|---|---|---|---|
| 1. Introduction & Consent | Establish legitimacy; obtain informed consent; set expectations | Who you are, purpose of study, estimated completion time, confidentiality assurance, voluntary participation statement, consent checkbox, contact information for questions/complaints | ~100 words |
| 2. Screening Questions | Ensure respondent meets inclusion criteria | Questions that determine eligibility: "Do you currently use mobile banking?" "Have you purchased from this website in the last 6 months?" — if No, the survey ends or redirects | 1–3 items |
| 3. Warm-Up / Easy Questions | Build respondent engagement; establish that the survey is manageable | Simple, non-threatening factual or behavioural questions: frequency of product use, years of experience, basic preferences | 3–5 items |
| 4. Core Construct Items | Measure the constructs in your conceptual framework | Your main Likert-scale or rating items, organised by construct (all items for Construct A, then all items for Construct B, etc.). Group related items together — don't mix constructs randomly. | 15–30 items (3–5 per construct × 3–6 constructs) |
| 5. Demographics | Collect variables for descriptive statistics and subgroup analysis | Age (ranges, not exact), gender, education, income, location (city/tier), industry, job role, years of experience, organisation size. Place demographics at the END — respondents are more likely to complete them after investing time in the core items. | 5–10 items |
| 6. Closing | Thank the respondent; provide debriefing or next steps | Thank you message; information about how results will be used; optional: invitation to participate in follow-up interview (if mixed-methods), link to enter incentive draw | ~50 words |
The ideal survey takes 8–12 minutes to complete. Beyond 15 minutes, response quality degrades significantly — respondents speed through items, select neutral midpoints, or abandon the survey entirely. For a capstone, aim for 25–35 items total (including demographics). If your survey exceeds 40 items, cut construct items or dimensions — you are likely trying to measure too many things. One well-measured construct is worth more than five poorly measured ones.
3. Sampling Strategies — Who, How Many, and How Selected
Sampling is the process of selecting a subset of a population to study. The goal is to draw conclusions about the population based on the sample — and the quality of those conclusions depends entirely on how the sample was selected. A perfectly designed questionnaire administered to a poorly selected sample produces perfectly measured noise.
3.1 Probability vs. Non-Probability Sampling
| Dimension | Probability Sampling | Non-Probability Sampling |
|---|---|---|
| Core Principle | Every element in the population has a known, non-zero probability of being selected. Selection is random. | Elements are selected based on convenience, judgement, or purpose. Selection probability is unknown. |
| Generalisability | Results can be statistically generalised from sample to population (with quantifiable margin of error) | Results cannot be statistically generalised. They may be analytically or theoretically transferable — but this requires argument, not statistics. |
| When to Use | When your RQs require population-level estimates, when you need to test hypotheses about prevalence or relationships with known confidence intervals, when a sampling frame exists | When your population is hard to reach or define, when you need specific types of participants (experts, rare experiences), when the research is exploratory, when probability sampling is infeasible within capstone constraints |
| Key Requirement | A complete, accurate sampling frame (list of all population elements) | A clear, defensible rationale for why the selected participants are appropriate for the RQs |
3.2 Probability Sampling Techniques
Every element has an equal probability of selection. Requires a complete sampling frame. Use a random number generator to select from the frame. Strengths: Unbiased; sampling error easily calculated. Weaknesses: Requires a complete list of the population; may miss small subgroups. Capstone Example: Randomly selecting 200 employees from an HR database of 2,000.
Divide the population into mutually exclusive strata (groups) based on a relevant characteristic, then randomly sample within each stratum. Strengths: Ensures representation of subgroups; more precise estimates than SRS for the same sample size. Weaknesses: Requires knowledge of strata proportions in the population; more complex to execute. Capstone Example: Stratifying by department (Marketing, Finance, Operations, IT) and randomly sampling proportionally within each.
Divide the population into clusters (naturally occurring groups — schools, companies, cities), randomly select clusters, then study all or a random sample of elements within selected clusters. Strengths: Practical when the population is geographically dispersed; reduces travel and administrative costs. Weaknesses: Higher sampling error than SRS (elements within a cluster tend to be similar). Capstone Example: Randomly selecting 10 colleges in a city, then surveying all final-year students in those colleges.
Select every k-th element from a list after a random start. Strengths: Simpler to execute than SRS; ensures spread across the frame. Weaknesses: Can introduce bias if the list has periodic patterns (every 7th entry is a different type). Capstone Example: Selecting every 15th customer from a CRM database of 3,000, starting from a randomly chosen number between 1 and 15.
3.3 Non-Probability Sampling Techniques — When and How to Use Them
| Technique | How It Works | Appropriate When... | Write in Your Methodology |
|---|---|---|---|
| Purposive Sampling | Deliberately select participants who meet specific criteria relevant to your RQs | You need participants with specific characteristics, experiences, or expertise that the general population does not possess | "Participants were selected through purposive sampling based on the following criteria: (a) minimum 5 years of experience as a product manager in Indian SaaS companies, (b) direct involvement in at least one product pricing decision in the last 12 months. Participants were identified through LinkedIn professional networks and the researcher's institutional alumni database." |
| Quota Sampling | Set quotas for subgroups (e.g., 50 men, 50 women; 30 urban, 30 semi-urban, 30 rural), then use convenience to fill each quota | You want to ensure representation of specific subgroups but probability sampling is infeasible | "Quota sampling was employed to ensure representation across organisation sizes: micro (1–10 employees), small (11–50), and medium (51–250), with a target of 40 responses per quota. Within each quota, participants were recruited through snowball sampling starting from the researcher's professional contacts." |
| Snowball Sampling | Start with a few participants who meet your criteria; ask them to refer others who also meet the criteria | Your population is hard to access or identify through conventional means (e.g., gig economy workers, angel investors, users of a specific niche technology) | "Given the absence of a sampling frame for gig economy workers in the target city, snowball sampling was used. Initial participants (n = 8) were identified through gig platform forums; each was asked to refer 2–3 other workers. The chain continued until the target sample size (n = 40) was reached." |
| Convenience Sampling | Select participants who are easily accessible — students in your class, people in your network, social media followers | You are conducting exploratory or pilot research, or you have exhausted all other feasible options and convenience is your only realistic approach | Be honest: "This study employed a convenience sample of undergraduate students at [institution]. This limits the generalisability of findings to similar populations. Results should be interpreted as indicative rather than representative. Future research should replicate with a probability sample." Never present a convenience sample as if it were random or representative. |
3.4 How Large Should Your Sample Be?
There is no single "correct" sample size — it depends on your analytical method, population characteristics, desired precision, and practical constraints.
| Analytical Method | Rule of Thumb | Formal Approach |
|---|---|---|
| Descriptive statistics only | n ≥ 100 for reasonable precision; n ≥ 385 for population-level estimates with 5% margin of error at 95% confidence (assuming large population) | Use a sample size calculator for proportions: n = Z²p(1−p)/e² where Z = 1.96 (95% CI), p = 0.5 (maximum variability), e = desired margin of error |
| Correlation / Regression | n ≥ 50 + 8k where k = number of predictors (Green, 1991). For 5 predictors: n ≥ 90. For multiple regression, n ≥ 100–150 is commonly recommended. | Power analysis using G*Power software: specify effect size (f² = 0.15 for medium), α = 0.05, power = 0.80, number of predictors → software calculates required n |
| t-test (two groups) | n ≥ 30 per group for Central Limit Theorem to apply; n ≥ 64 per group to detect a medium effect (d = 0.5) with power = 0.80 | Power analysis: specify test (independent t-test), effect size (d), α, power → software calculates required n per group |
| ANOVA (3+ groups) | n ≥ 30 per group minimum; more for small expected effects or unequal group sizes | Power analysis: specify test (one-way ANOVA), effect size (f), α, power, number of groups → software calculates required n per group |
| Structural Equation Modelling (SEM) | n ≥ 200 minimum; n ≥ 10 per estimated parameter; n ≥ 20 per construct with multiple indicators | Monte Carlo simulation; rules of thumb vary widely (n = 100–400+) depending on model complexity, number of indicators per factor, and communality levels |
Most capstone samples will not meet the formal requirements for probability sampling — and that is acceptable if you are honest about it. A convenience sample of 120 respondents analysed with regression, where you acknowledge that "generalisability is limited by the non-probability nature of the sample and findings should be considered exploratory," is methodologically honest. A convenience sample of 120 presented as if it were representative of "Indian consumers" is methodologically dishonest. The panel and supervisor will accept practical constraints. They will not accept misrepresentation.
4. Experimental & Quasi-Experimental Designs
Experiments are the gold standard for establishing causality — determining whether X causes Y, not just whether X and Y are correlated. They are less common in BBA capstones than surveys (requiring manipulation and control) but are increasingly feasible with online platforms, and are the dominant quantitative method for BCA capstones involving system or algorithm evaluation.
4.1 The Experimental Design Spectrum
| Design Type | Key Features | Threats to Validity Controlled | BBA Example | BCA Example |
|---|---|---|---|---|
| True Experiment (RCT) | Random assignment to conditions; manipulation of IV; control group; high internal validity | Selection bias (randomisation); history and maturation (control group); testing effects (control group receives same pre/post tests) | Effect of pricing format (₹999 vs. ₹1,000) on purchase intention — 200 participants randomly assigned to one of two conditions, shown the same product with different price displays | Effect of code review method (manual checklist vs. AI-assisted tool) on bug detection rate — 30 developers randomly assigned to conditions, reviewing the same code sample |
| Quasi-Experiment | Manipulation of IV but NO random assignment; uses pre-existing groups or natural interventions; moderate internal validity | Maturation and history (comparison group); selection bias partially addressed through matching or statistical control | Effect of a mandatory CSR reporting regulation (SEBI, 2021) on firm profitability — comparing treatment group (top 1,000 listed firms, subject to regulation) with comparison group (next 1,000 firms, not subject) | Effect of adopting DevOps practices on deployment frequency — comparing teams that adopted DevOps in 2024 (treatment) with teams of similar size and domain that did not (comparison) |
| Pre-Experimental | One group, pre-test/post-test or post-test only; no control group; weak internal validity | Very few — cannot rule out history, maturation, testing, or regression to the mean as alternative explanations | Measuring employee satisfaction before and after a team-building intervention on one team — any change could be due to the intervention OR to other events, natural improvement, or regression | Measuring page load time before and after a code optimisation on one module — improvement could be due to the optimisation OR to network conditions, server load, or other concurrent changes |
4.2 The Validity Framework for Experimental Designs
Question: Can we be confident that the IV (and not something else) caused the observed change in the DV? Threats: History (events between pre and post), maturation (natural change in participants), testing (practice effects), instrumentation (changes in measurement), selection bias, mortality (differential dropout), regression to the mean. Mitigation: Random assignment; control group; consistent measurement; monitoring and documenting external events.
Question: Do the findings generalise to other people, settings, times, and operationalisations? Threats: Interaction of selection and treatment (sample is not representative), interaction of setting and treatment (lab results don't transfer to field), interaction of history and treatment (findings are time-bound). Mitigation: Replicate with different populations and settings; describe sample and context in detail so readers can judge transferability; acknowledge boundary conditions.
Question: Does the operationalisation (measurement or manipulation) actually capture the theoretical construct? Threats: Mono-operation bias (only one way of measuring/manipulating), mono-method bias (all data from one method), confounding constructs (measuring something else), demand characteristics (participants guess the hypothesis). Mitigation: Use validated scales; employ multiple measures; pilot test manipulations; include manipulation checks.
Question: Are the statistical conclusions correct — is there really a relationship, and how strong is it? Threats: Low statistical power (sample too small), violated assumptions (non-normal data, heteroskedasticity), fishing and error rate inflation (multiple tests without correction), unreliability of measures (low Cronbach's alpha). Mitigation: Power analysis before data collection; check and report assumption tests; correct for multiple comparisons (Bonferroni, Holm); ensure instruments have α ≥ 0.70.
There is an inherent tension between internal and external validity. Tightly controlled lab experiments maximise internal validity but may not generalise to real-world settings (low external validity). Field studies in natural settings have high external validity but many uncontrolled variables threaten internal validity. The researcher's job is not to eliminate all threats to validity — that is impossible. The job is to: (a) identify the most serious threats given your design, (b) mitigate them where possible, and (c) acknowledge those that remain as limitations. A study with acknowledged limitations is methodologically honest. A study that claims to have no validity threats is methodologically naive.
5. Secondary Data Analysis
Secondary data analysis — using data collected by someone else for a different purpose — is an increasingly viable option for capstone projects. It eliminates the time and resource burden of primary data collection, often provides larger and more representative samples, and for BBA students in particular, enables analysis of data (financial, economic, demographic) that would be impossible to collect independently.
5.1 Common Secondary Data Sources for Capstone Research
| Category | Examples | Access | Typical Use in Capstone |
|---|---|---|---|
| Government Statistical Agencies | NSSO (consumption, employment), RBI (banking, financial inclusion), MOSPI (economic indicators), Census of India | Mostly public; unit-level data may require application | BBA: studying consumption patterns, financial inclusion trends, demographic analysis. BCA: studying digital divide, technology access patterns. |
| Financial Databases | ProwessIQ (Indian firms), Bloomberg Terminal, Refinitiv Eikon, CMIE (multiple databases) | Institutional subscription required; check your library | BBA: financial ratio analysis, ESG disclosure scoring, corporate governance research. Limited BCA application. |
| Open Data Repositories | data.gov.in (Indian government open data), World Bank Open Data, IMF Data, WHO Global Health Observatory, Kaggle datasets | Free and public | Cross-country comparisons; time-series trend analysis; supplementing primary data with contextual indicators. |
| Academic Data Archives | ICPSR (social science), UK Data Service, Harvard Dataverse, Zenodo, paperswithcode.com (ML benchmarks) | Mostly free; registration required; some restricted-access | Replicating or extending published research; using established benchmark datasets for ML model evaluation. |
| Platform & Web Data | GitHub Archive, Stack Overflow Developer Survey, Stack Exchange Data Explorer, Google Trends, Wikipedia pageview statistics | Free via APIs or periodic data dumps | BCA: software engineering research, developer behaviour analysis, technology adoption trends. BBA: consumer interest tracking via Google Trends. |
| Organisational Data | Company records, CRM databases, HR systems, transaction logs, web analytics | Requires organisational permission; NDA likely required | BBA: customer churn analysis, employee turnover study. BCA: system performance analysis, user behaviour modelling. Requires formal data-sharing agreement and ethics approval. |
5.2 Evaluating Secondary Data Quality — The DA FIBS Framework
| Criterion | Questions to Ask | Red Flags |
|---|---|---|
| Documentation | Is there a codebook or data dictionary? Are variables clearly defined? Is the data collection methodology documented? Are known issues or limitations reported? | No documentation; variables with unclear definitions; no information on missing data; no sample design description |
| Appropriateness | Do the variables operationalise the constructs in your conceptual framework? Is the population and time period relevant to your RQs? Is the level of analysis (individual, firm, country) appropriate? | Variables that approximate but don't match your constructs; data from a different population or time period than your RQs require; wrong unit of analysis |
| Fidelity | How reliable are the measurements? If survey-based: what was the response rate? If sensor-based: what is the measurement error? Are there known biases in the data collection? | Very low response rates (<10%); self-reported data on sensitive topics without validation; known measurement errors that are not corrected |
| Integrity | Are there missing values? Are there outliers or impossible values? Is the data internally consistent (e.g., sub-categories sum to totals)? Has the data been manipulated or cleaned in undocumented ways? | Large proportions of missing data (>20% for key variables); impossible values (negative ages, percentages >100); inconsistencies; evidence of undocumented cleaning |
| Boundaries | Who collected this data and why? What was the original purpose? Are there restrictions on how the data can be used (licence, terms of use, ethics constraints)? | Unknown provenance; data collected for a purpose that conflicts with your use; restrictive licences; ethics concerns about the original data collection |
| Sustainability | Is the dataset still maintained and updated? Can you access it for the duration of your capstone? Is there a permanent identifier (DOI) for citation? | Dataset no longer maintained; access expected to expire during your capstone; no permanent identifier or version number |
5.3 Common Pitfalls in Secondary Data Analysis
| Pitfall | What It Looks Like | How to Avoid |
|---|---|---|
| Variable Forcing | "The dataset has 'number of training hours' — I'll use that as a proxy for 'organisational learning culture.'" — Proxying a construct with whatever variables happen to be available, regardless of fit. | Start with your RQs and conceptual framework, then find data. Not: start with an available dataset and find RQs that fit. If the dataset cannot adequately operationalise your constructs, it is the wrong dataset. |
| Ignoring the Original Design | Analysing survey data without reading the sampling methodology, so you don't know that the sample was stratified by region — and treating it as a simple random sample produces incorrect standard errors. | Read the technical documentation. Understand the sampling design, weighting, and survey methodology before you run a single analysis. Complex survey designs require specialised analytical techniques (survey-weighted regression, not OLS). |
| Data Dredging | Running correlations between every variable in a 200-variable dataset, reporting the 5 that are significant, and building a narrative around them post-hoc. | Pre-specify your hypotheses and analytical plan before exploring the data. Treat exploratory analyses as exploratory — label them clearly. Correct for multiple comparisons if conducting many tests. |
| Missing Data Mismanagement | Deleting all rows with any missing value (listwise deletion) without checking whether the missing data is random or systematic, potentially biasing results. | Report the extent and pattern of missing data. Use appropriate methods: multiple imputation (MCAR/MAR assumptions), sensitivity analysis, or acknowledge that listwise deletion may bias results if data is not MCAR. |
| Black Box Provenance | "I downloaded this dataset from Kaggle. I don't know who collected it, how, when, or why." — The methodology chapter cannot describe the data's origin because the student doesn't know it. | Only use datasets whose provenance you can document. If you cannot write a paragraph describing who collected the data, when, how, and for what purpose, you cannot use that dataset for a capstone. This rules out many Kaggle datasets with no documentation. |
Think Deeper — Cross Questions
Discuss in pairs before sharing with the class.
You are designing a survey on "employee well-being" for your capstone. You find a validated 24-item scale from a top-tier journal. Using the full scale would make your questionnaire 40+ items including demographics — likely exceeding the 15-minute completion target. What do you do? What are the trade-offs between using the full validated scale (measurement quality) and shortening it (response quality)? Is it legitimate to select a subset of items from a validated scale?
A BCA student proposes to evaluate their machine learning model using a dataset they found on GitHub with no licence, no documentation, and no information about how the data was collected or labelled. Their supervisor suggests they use an established benchmark dataset instead. The student argues the GitHub dataset is "more realistic." Analyse this disagreement. What are the ethical, methodological, and practical concerns with using undocumented data in a capstone?
You have designed a true experiment with random assignment, but during data collection, 30% of participants in the treatment group drop out (they don't complete the post-test), while only 5% drop out from the control group. Your final analysis compares the remaining 70% of treatment participants with the remaining 95% of control participants. What threat to validity does this introduce? Is your study still a true experiment? What should you do — both analytically and in terms of reporting?
Two capstone students study "the impact of social media marketing on brand loyalty." Student A surveys 500 students in their university using a convenience sample, achieves a high response rate, and reports that "social media marketing significantly predicts brand loyalty among Indian consumers (β = 0.34, p < 0.001)." Student B surveys 120 smartphone users selected through stratified random sampling from a panel of 50,000, achieves a moderate response rate, and reports that "social media marketing is positively associated with brand loyalty in this sample (β = 0.29, p = 0.04); generalisability is limited to panel members." Critically compare these two approaches. Whose conclusions are more trustworthy — and why?
Quick Check — Questionnaire Item Diagnosis
Each item below is from a student-designed questionnaire. Diagnose the primary problem with the item.
1. "How satisfied are you with the quality, price, and customer service of our product?" [Scale: 1 = Very Dissatisfied to 5 = Very Satisfied]
2. "Don't you agree that the government should increase spending on healthcare infrastructure?" [Scale: 1 = Strongly Disagree to 5 = Strongly Agree]
3. "On average, how many hours per day do you spend using social media applications on your smartphone during a typical workday over the past three months?"
4. "What is your annual household income? (a) 0–3 lakhs (b) 3–6 lakhs (c) 6–9 lakhs (d) 9 lakhs and above"
Knowledge Check — Interactive Quiz
Test your understanding of quantitative research design.
Q1. Which of the following is the defining characteristic of a true experiment?
Q2. A questionnaire item asks: "The company's HR policies are fair and transparent." What is the main problem with this item?
Q3. A student uses convenience sampling to survey 200 classmates and reports: "78% of Indian consumers prefer online shopping over in-store shopping." What is the primary methodological problem?
Q4. In the validity framework, "history" is a threat to which type of validity?
Q5. For a multiple regression with 6 predictors, what is the minimum recommended sample size according to Green's (1991) rule of thumb?
Lab Activity — Developing Quantitative Instruments
Part A: Draft Your Survey Questionnaire or Data Extraction Protocol (60 min)
- Create your construct map (Section 2.1, Step 1). For each construct in your conceptual framework: define it conceptually, identify its dimensions, and specify the number of items planned. If using existing scales, cite the source.
- Draft your items following the 8 rules in Section 2.2. For each item, note whether it is adopted (cite source), adapted (cite original + explain adaptation), or self-created (justify why no existing scale was suitable).
- Structure your questionnaire following the 6-section flow in Section 2.3. Write the introduction and consent section. Order your items: screening → warm-up → core constructs → demographics → closing.
- Apply the item diagnosis check from the Quick Check exercise to each of your items. Fix any problems before the pretest.
| Construct | Conceptual Definition | Dimensions | Item | Source | Response Scale |
|---|---|---|---|---|---|
| Example: Perceived Usefulness (TAM) | Degree to which a person believes that using a particular system would enhance their job performance (Davis, 1989) | Performance enhancement; efficiency; productivity | "Using this app would improve my performance in managing daily tasks." | Adopted: Davis (1989), item PU2 | 5-point Likert: 1=Strongly Disagree, 5=Strongly Agree |
| (Your construct 1) | |||||
| (Your construct 2) | |||||
| (Your construct 3) |
For BCA students not using surveys: If your capstone involves evaluating a system, algorithm, or model, your "instrument" is your evaluation protocol. Adapt the template: constructs become evaluation criteria (accuracy, efficiency, usability, fairness); items become specific metrics (F1 score, inference time in ms, System Usability Scale, demographic parity difference); and the response scale becomes the measurement method.
Part B: Design Your Sampling Plan (45 min)
- Define your target population precisely. Not "consumers" but "smartphone users aged 18–35 in Tier-1 Indian cities who have made at least one online purchase in the last 6 months."
- Identify your sampling frame. Where will you find your participants? Is there a list (sampling frame)? If not, how will you access the population?
- Choose your sampling technique (Section 3.2 or 3.3). Justify your choice with reference to feasibility, population characteristics, and analytical requirements. If using non-probability sampling, acknowledge the generalisability limitation explicitly.
- Calculate your target sample size using the guidelines in Section 3.4. Document your calculation. Add 20–30% to account for non-response and incomplete responses (e.g., if you need n = 100, target n = 125–130).
- Write your sampling section for your methodology chapter — 200–300 words that another researcher could follow to replicate your sampling procedure.
Part C: Secondary Data Exploration (45 min)
Even if your primary method is survey or experiment, identify at least one relevant secondary dataset that could contextualise or supplement your findings.
- Search for a relevant dataset using the sources in Section 5.1. For BBA: check ProwessIQ, RBI DBIE, data.gov.in, World Bank. For BCA: check Kaggle, paperswithcode, GitHub Archive, Stack Overflow Developer Survey.
- Evaluate the dataset using the DA FIBS framework (Section 5.2). Complete the evaluation table.
| Criterion | Your Assessment | Rating (✓/✗/?) |
|---|---|---|
| Documentation | ||
| Appropriateness | ||
| Fidelity | ||
| Integrity | ||
| Boundaries | ||
| Sustainability |
Exit Ticket
Submit with your draft instrument and sampling plan.
- Submit your draft questionnaire or evaluation protocol (Part A). Which item are you least confident about, and why?
- What is your target sample size and how did you calculate it? What sampling technique will you use, and why?
- Identify the most significant threat to validity in your quantitative design. What is your strategy for mitigating it?
- If using primary data: When will you pretest your instrument, and with whom? If using secondary data: What is the DA FIBS rating of your primary dataset?
- One specific question about quantitative research design that you need answered before you proceed to data collection:
Key Takeaways — Week 9
A survey is not a list of interesting questions — it is a measurement instrument, as much as a thermometer or a voltmeter. Every item must be justified, tested, and linked to a construct. Adopt validated scales before creating your own items. Pretest before deploying.
Probability sampling enables statistical generalisation. Non-probability sampling does not — and pretending otherwise is the most common methodological error in capstone research. Honesty about the limits of your sample is a sign of methodological competence, not weakness.
Every research design faces threats to internal, external, construct, and statistical conclusion validity. The goal is not a threat-free design (impossible) but a design where the most serious threats are identified, mitigated where possible, and acknowledged where not.
Your methodology chapter must enable replication. Another researcher reading your methodology should be able to reproduce your study — same population, same instrument, same procedures. If they cannot, your methodology section is incomplete. This standard applies equally to primary and secondary data.
Facilitator Notes
Preparation Checklist
- Prepare 2–3 examples of real published questionnaires (one BBA domain, one BCA evaluation protocol) to show as exemplars. Highlight the construct-to-item mapping, the response scale choices, and the flow from introduction to demographics.
- Prepare the "bad survey" for the questionnaire critique activity (Lecture 1 segment). Include at least 8 items with deliberate errors: double-barrelled, leading, double negative, overlapping categories, missing N/A option, overly complex, assumed knowledge, ambiguous. Students learn more from correcting errors than from seeing perfect examples.
- Have G*Power software (free) available or recommend an online power analysis calculator (e.g., ClinCalc, StatPages). Walk through at least one power analysis live during the sampling lecture so students see how to use it.
- Identify which secondary databases your institution subscribes to (ProwessIQ, Bloomberg, etc.) and share access instructions. If access requires librarian approval, start the process before this week — students will need it by Week 11–12.
- For BCA cohorts: compile a list of 5–10 well-documented benchmark datasets relevant to common BCA capstone topics (NLP, computer vision, software engineering, recommender systems). Include the dataset name, URL, citation, licence, and a one-line description.
Common Student Difficulties
- Starting from scratch instead of adopting: Students believe creating their own items demonstrates originality. Correct this firmly: originality belongs in the RQs and contribution, not in measurement. Adopting validated scales is methodologically superior to creating novel items, and far more efficient. The capstone is not a scale development exercise.
- Confusing questionnaire length with rigour: "I have 85 items because I want to be thorough." An 85-item questionnaire produces respondent fatigue, high dropout, and low-quality data. Enforce the 40-item ceiling. If a student insists they need more items, ask them to justify each item's contribution to answering a specific RQ. Most cannot.
- Avoiding sample size calculation: Students write "sample size will be 100" without any calculation or justification. Require every student to document how they arrived at their sample size — even if the answer is "I used Green's (1991) rule for regression with 4 predictors: n ≥ 50 + 8×4 = 82, rounded up to 100." The calculation matters more than the specific number.
- Presenting convenience samples as representative: "A survey of 150 university students revealed that Indian consumers prefer..." — this is the single most common methodological error in BBA capstones. Flag it every time you see it. Require students to explicitly state the limits of their sample's generalisability in their methodology section.
- The DA FIBS framework feels abstract: Students complete the checklist mechanically without understanding the implications. Bring in a real secondary dataset with documentation problems and walk through the evaluation together. Show what happens when you analyse data whose provenance you don't understand.
Pacing Tips
- The questionnaire critique activity (pairs, 25 min) is one of the most effective segments — don't rush it. Give students a poorly designed survey and 15 minutes to find as many problems as they can. Then debrief as a class, item by item. The discussion reveals that item-writing errors are not obvious to novices but become glaring once pointed out.
- The sampling lecture (Lecture 2) is dense. Consider splitting it: probability techniques and sample size calculation in the main lecture; non-probability techniques as a shorter, focused segment with more examples relevant to students whose populations lack sampling frames (common in capstone research).
- Some students will be overwhelmed by the transition from proposal to instrument development. Reassure them: the survey they draft this week is a first draft. It will be revised after pretesting, supervisor feedback, and (for some) peer review. The goal this week is a complete draft, not a perfect instrument.
- Part C (secondary data) is valuable even for students doing primary data collection. Knowing what secondary data exists contextualises primary findings and provides fallback options if primary data collection encounters delays.