Understanding Survey Results

The total count of respondents in an institution's raw data file will differ from the numbers reported in the institutional reports due to intentional exclusion of certain surveys in the online reporting system. Respondents are excluded from the institutional reports for the following reasons.

• The respondent did not indicate full-time or part-time enrollment status at the institution. (Variable: ENRLMENT is blank)
• The respondent did not indicate their gender identity. (Variable: GENDER_4CAT is blank)
• The respondent did not indicate entering or returning student status. (Variable: TERMSENR is blank)
• The survey is invalid. A survey is invalid if a student does not answer any of the 19 sub-items in Item 19, answers four or more times to all 19 sub-items, or answers never to all 19 sub-items. (Variables: ASKQUES, PREPDRFT, LATETURN, NOTTURN, SUPINSTR, NOTCOMPL, PINCLASS, PREPOUTC, GRPSTUDY, NRGSTUDY, USEINTMG, MAILFAC, FACASSN, CLASSREL, FEEDBACK, RCVGRDS, FACIDOC, OCIDEAS, SKIPCL)
• The respondent indicated their age as under 18. Note: Data for these students are not returned. (Variable: AGENEW is 1. Responses where age was not marked are included in the analysis.)
• The respondent indicated they had taken the survey in a previous class or did not respond to item 1. (Variable: SRVAGAIN is 1 or blank)
• Oversampled respondents are not included because they are selected outside of SENSE's primary sampling procedures. (Variable: PSAMPLE is 0) *not applicable to online administration
• The respondent indicated that they were a returning student (Variable: TERMSENR is not 1)

SENSE benchmarks are groups of conceptually related survey items that focus on institutional practices and behaviors that promote engagement among entering students. Benchmarks are used to compare each institution's performance to that of similar institutions and to  the SENSE Cohort. The six SENSE benchmarks are: early connections, high expectations and aspirations, clear academic plan and pathway, effective track to college readiness, engaged learning, and academic and social support network.

Benchmark reports consist of tables showing the college's scores on each benchmark, followed by means and frequency tables of items in each benchmark. Benchmark scores provide an overview of how the college is doing in particular areas. Colleges must be mindful that the individual survey items composing each benchmark deserve examination.

Benchmarks Over Time
The benchmark reports also include a graph that displays raw benchmark scores over the college's last five administrations, or up to the last five administrations. The raw benchmarks over time graph can be found on the third tab (Raw Benchmarks & Graphs) of the Benchmark Excel files in Standard Reports.

How Benchmarks Are Calculated

Responses to individual SENSE survey items are summarized in two formats: means and frequencies.

Means reports present an average for each survey item that has scaled responses (e.g., strongly agree to strongly disagree) and compare average item responses between member colleges (e.g., similarly sized colleges) and various groups, or between subgroups within a college (e.g, first-generation and not-first generation). Means are not run on dichotomous items (those with only two response options). These items are summarized in frequency reports.

Frequency reports present the observed frequencies of occurrence (counts and percentages) of the values for each survey item (excluding demographic survey items). These reports are useful for understanding how data are distributed across response categories. Please note that counts and percentages on frequency reports are subject to rounding.

When interpreting mean differences across comparison groups, CCCSE uses a combination of two measures: (1) a t-test with a very conservative alpha level of .001 or less (to determine if the difference between two means is significant and not likely due to chance) and (2) an effect size of .20 (absolute value) or more using Cohen’s d (to show the magnitude of difference between the two means). If a comparison meets both of these criteria, then it is considered to be a statistically significant difference worthy of further investigation. Comparisons that meet these criteria are marked with a double-asterisk (**). For internal analysis of small groups, it may make sense for colleges to use a larger alpha level but typically not a larger effect size.

In SENSE sampling procedures, students are sampled at the classroom level. As a result, full-time students, who by definition are enrolled in more classes than part-time students, are more likely to be sampled. This introduces a sampling bias resulting in disproportionately more full-time students completing the survey. Statistical weighting, discussed in detail below, is used to adjust for this bias.

In order to boost responses to the online survey, sampling is not employed; the survey invitation is sent to all eligible students. Research shows that full-time students and women are more likely to complete online surveys than part-time students and men. As a result, full-time students and women are overrepresented in the final sample. Statistical weighting, discussed in detail below, is used to adjust for this bias.

Beginning with SENSE 2022, CCCSE staff began using the IPEDS counts for degree/certificate-seeking first-time and degree/certificate-seeking transfer-in fall enrollment instead of total fall enrollment. This change should provide more accurate weights for the SENSE population. One additional change to weights for SENSE 2022 is based on research that has found that full-time students and women are more likely to complete online surveys than part-time students and men. The national distribution of survey respondents reflects this pattern. Therefore, weights are now based on both enrollment status and gender identity. These new weights are applied to both the paper-and-pencil and online survey results. The use of the new weights will not negatively impact analysis of the paper-and-pencil surveys. For example, if the proportion of men and women among survey respondents exactly matches the proportion in the institution population, results based on the new weights would be the same as results based on enrollment status alone. The weight variable in the download data set is still named IWEIGHT. When conducting analyses in which either enrollment status or gender identity is included as a breakout variable, weights should not be applied.

The online SENSE administration may result in different survey participation patterns by student characteristics (e.g., enrollment status, gender identity) compared to an in-class administration. Therefore, CCCSE staff encourages you to closely review the student characteristics of the survey respondents compared to the characteristics of your college’s entire student population. If there are substantial differences between the two groups, calculating weights that specifically take into account these differences may be useful in analyzing your raw data file. CCCSE’s research team is available to assist with this task. Contact data@cccse.org for guidance and/or assistance.

Item 2: “Thinking about this semester/quarter, how would you describe your enrollment at this college?”

Three sub-items in Item 17: “In which of the following types of courses were you enrolled during your first semester/quarter at this college?”

17a. Developmental Reading (also referred to as Basic Skills, College Prep, etc.)
17b. Developmental Writing (also referred to as Basic Skills, College Prep, etc.)
17c. Developmental Math (also referred to as Basic Skills, College Prep, etc.)

If a student answered YES to any of the three items above, they are classified as developmental. If they answer YES to any item, they do not need to provide a response to the other two. To be classified as non-developmental, a student must answer NO to all three of the items. If a student answers NO to only one or two items and does not provide a response to the other item(s), their developmental status is coded as missing.

Item 30: “Mark your age group.”

Respondents under age 18 are excluded from all data sets. Respondents marking age groups 18 to 19, 20 to 21, and 22 to 24 are classified as Traditional-Age and those marking age groups 25 to 29, 30 to 39, 40 to 49, 50 to 64, or 65+ are classified as Nontraditional-Age.

Item 39: “Who in your family has attended at least some college?”

If respondents indicated that their mother or father had attended at least some college, then those students are classified as Not First-Generation. If respondents did not indicate that their mother or father attended at least some college but selected any other response option (including “no one”), they are classified as First-Generation. If respondents did not select any response options, their first-generation status is classified as missing.

Note: Prior to the 2023 administration, if respondents did not select any response options, they were classified as first-generation.

Item 29: “Your gender identity.”

If the respondent indicated “Man,” “Woman,” or “I prefer not to respond,” the student’s Gender Identity status is coded as their respective choice. If respondent indicated “I prefer to self-identify,” their Gender status is coded as missing.

Note: Prior to the 2024 administration, this was the SEX variable:
Item 29: “Your sex” (See survey)

If respondent indicated “Male,” they are classified as “Man.” If respondent indicated that they are “Female,” they are classified as “Woman.”

Note: Students taking the online survey were allowed to indicate “I prefer not to respond.” This response option was not available on the paper survey.

For the 2024 administration and beyond:
Item 29: “Your gender identity.”

Prior to the 2024 administration:
This variable did not exist and colleges will not have any data for this variable

For the 2024 administration and beyond:
Item 36: “What is your racial or ethnic identification? (Mark all that apply)”

Race/Ethnicity is coded as “I prefer not to respond” if a student selected this response option, regardless of whether another response was also marked.

If a student selected more than one race/ethnicity response options, the race/ethnicity variable is set to “Two or More Races.”

Prior to the 2024 administration:
This variable did not exist, and colleges will not have any data for this variable

For the 2024 administration and beyond:
Item 36: “What is your racial or ethnic identification? (Mark all that apply)”

Race/Ethnicity is coded as American Indian or Native American if the respondent selected the American Indian, Indigenous, Native American, or Alaskan Native race/ethnicity response option.

Race/Ethnicity is coded as Asian, Asian American, or Pacific Islander if a student selected either the Asian or Pacific Islander (non-Native Hawaiian) race/ethnicity response options.

Race/Ethnicity is coded as Native Hawaiian if a student selected the Native Hawaiian race/ethnicity response option.

Race/Ethnicity is coded as Black or African American, Non-Hispanic if a student selected the Black or African American race/ethnicity response option.

Race/Ethnicity is coded as White, Non-Hispanic if a student selected the White race/ethnicity response option.

Race/Ethnicity is coded as Hispanic, Latino, Spanish if a student selected the Hispanic or Latino race/ethnicity response option.

Race/Ethnicity is coded as Other if a student selected the “Other” race/ethnicity response option.

Race/Ethnicity is coded as missing if a student selected “I Prefer Not to Respond” or if a student selected more than one race/ethnicity response option.

Note: Prior to the 2024 administration, (formerly the DIVERSIT variable):
Item 35: “What is your racial/ethnic identification?” (See survey)

Online Survey Item: “During the current academic term, how many classes are you taking...(a) Face-to-face, (b) Online, (c) Hybrid”

If the respondent took the survey on paper (in a classroom) or indicated taking at least one face-to-face or hybrid class, they are categorized as “Not Online-Only.”

Item 35: “Did you receive a federal Pell Grant as part of your financial aid package at this college this year?”

If respondents indicated that they received a federal Pell Grant, those students are classified as Pell Grant Recipients. If respondents indicated that they did not receive a federal Pell Grant, those students are classified as Non-Pell Grant Recipients

If a student took the survey on paper in a randomly selected classroom, they are classified as In-Class Survey. If a student took the online version of the survey, they are classified as Online Survey.