The evaluation of the toxicity of magnetic nanoparticles (MNPs) has attracted much attention in recent years. The current study aimed to investigate the cytotoxic effects of Fe3O4, oleic acid-coated Fe3O4 (OA-Fe3O4), and carbon-coated Fe (C-Fe) nanoparticles on human hepatoma BEL-7402 cells and the mechanisms. WST-1 assay demonstrated that the cytotoxicity of three types of MNPs was in a dose-dependent manner. G1 (Fe3O4 and OA-Fe3O4) phase and G2 (C-Fe) phase cell arrests and apoptosis induced by MNPs were detected by flow cytometry analysis. The increase in apoptosis was accompanied with the Bax over-expression, mitochondrial membrane potential decrease, and the release of cytochrome C from mitochondria into cytosol. Moreover, apoptosis was further confirmed by morphological and biochemical hallmarks, such as swollen mitochondria with lysing cristae and caspase-3 activation. Our results revealed that certain concentrations of the three types of MNPs affect BEL-7402 cells viability via cell arrest and inducing apoptosis, and the MNPs-induced apoptosis is mediated through the mitochondrial-dependent pathway. The influence potency of MNPs observed in all experiments would be: C-Fe > Fe3O4 > OA-Fe3O4.