Background: High glycemic variability (GV) is common in critically ill patients; however, the prevalence and mortality association with early GV in patients with sepsis remains unclear. Methods: This retrospective cohort study was conducted in a medical intensive care unit (ICU) in central Taiwan. Patients in the ICU with sepsis between January 2014 and December 2015 were included for analysis. All of these patients received protocol-based management, including blood sugar monitoring every 2 h for the first 24 h of ICU admission. Mean amplitude of glycemic excursions (MAGE) and coefficient of variation (CoV) were used to assess GV. Results: A total of 452 patients (mean age 71.4 ± 14.7 years; 76.7% men) were enrolled for analysis. They were divided into high GV (43.4%, 196/452) and low GV (56.6%, 256/512) groups using MAGE 65 mg/dL as the cut-off point. Patients with high GV tended to have higher HbA1c (6.7 ± 1.8% vs. 5.9 ± 0.9%, p < 0.01) and were more likely to have diabetes mellitus (DM) (50.0% vs. 23.4%, p < 0.01) compared with those in the low GV group. Kaplan–Meier analysis showed that a high GV was associated with increased 30-day mortality (log-rank test, p = 0.018). The association remained strong in the non-DM (log-rank test, p = 0.035), but not in the DM (log-rank test, p = 0.254) group. Multivariate Cox proportional hazard regression analysis identified that high APACHE II score (adjusted hazard ratio (aHR) 1.045, 95% confidence interval (CI) 1.013–1.078), high serum lactate level at 0 h (aHR 1.009, 95% CI 1.003–1.014), having chronic airway disease (aHR 0.478, 95% CI 0.302–0.756), high mean day 1 glucose (aHR 1.008, 95% CI 1.000–1.016), and high MAGE (aHR 1.607, 95% CI 1.008–2.563) were independently associated with increased 30-day mortality. The association with 30-day mortality remained consistent when using CoV to assess GV. Conclusions: We found that approximately 40% of the septic patients had a high early GV, defined as MAGE > 65 mg/dL. Higher GV within 24 h of ICU admission was independently associated with increased 30-day mortality. These findings highlight the need to monitor GV in septic patients early during an ICU admission.