Organically modified rectorite (OREC) micro/nanoparticles can be synthesized by organic modification from calcium rectorite (Ca(2+)-REC or REC), a common form of rectorite in nature. Although REC and OREC have potential applications in food packing and drug delivery, their cytotoxicity is not clear. In the present study, we investigated and compared the cytotoxicity of REC and OREC micro/nanoparticles in Chang liver cells, the human normal hepatic cells, and human hepatoma HepG2 cells. The interlayer spacing of OREC was enlarged after organic modification. After treatment with REC or OREC for 24 h at 1 and 5 μg/mL, they were taken up by Chang liver cells. REC and OREC induced cytotoxicity in Chang liver and HepG2 cells at almost all doses (1, 2.5, 5, 7.5, and 10 μg/mL) after 6, 24, and 48 h of treatment (P < 0.05 or P < 0.01). Compared with REC, OREC was more cytotoxic. However, there was no difference in the cytotoxicity of REC and OREC between the two cell lines. After treatment with REC or OREC at 7.5 and 10 μg/mL for 24 h, the apoptotic and necrotic percentages of Chang liver cells were increased (P < 0.05 or P < 0.01). The levels of apoptosis-related proteins Bax, Bcl-2, and pro-caspase-3 were all decreased in Chang liver cells after 24 h of exposure to REC or OREC at 5, 7.5, 10 μg/mL. There was no change in the relative ratio of Bax/Bcl-2 after treatment, indicating that REC or OREC-induced apoptosis was not associated with Bax-related mitochondria-mediated apoptotic pathway. Our results suggested that OREC was more cytotoxic than REC, but the underlying mechanisms need further investigation.