Abstract Background PEP02 (also known as MM-398, nal-IRI) is a novel nanoparticle formulation of irinotecan encapsulated in liposomes. The aims of this study were to investigate the dose-limiting toxicity (DLT), maximum tolerated dose (MTD) and pharmacokinetics (PK) of PEP02 in combination with 5-FU and LV, in patients with advanced refractory solid tumors. Methods Patients were enrolled in cohorts to receive PEP02 from 60 to 120 mg/m2 (dose expressed as the irinotecan hydrochloride trihydrate salt) as a 90-min intravenous infusion on day 1, followed by 24 h infusion of 5-FU 2,000 mg/m2 and LV 200 mg/m2 on days 1 and 8, every 3 weeks. Results A total of 16 patients were assigned to four dose levels, 60 (three patients), 80 (six patients), 100 (five patients) and 120 mg/m2 (two patients). DLT was observed in four patients, two at the 100 mg/m2 dose level (one had grade III infection with hypotension and grade III hemorrhage; the other had grade III diarrhea and grade IV neutropenia), and two at the 120 mg/m2 dose level (one had grade III diarrhea and grade IV neutropenia; the other had grade III diarrhea). The MTD of PEP02 was determined as 80 mg/m2. The most common treatment-related adverse events were nausea (81%), diarrhea (75%) and vomiting (69%). Among the six patients who received the MTD, one patient exhibited partial response, four patients had stable disease and one showed progressive disease. Pharmacokinetic data showed that PEP02 had a lower peak plasma concentration, longer half-life, and increased area under the plasma concentration-time curve from zero to time t of SN-38 than irinotecan at similar dose level. Conclusions The MTD of PEP02 on day 1 in combination with 24-h infusion of 5-FU and LV on days 1 and 8, every 3 weeks was 80 mg/m2, which will be the recommended dose for future studies. Trial registration The trial was retrospectively registered ( NCT02884128 ) with date of registration: August 12, 2016.