Ishmael’s family – apparently fairly faithful and full of daughters to match Lehi’s sons, left Jerusalem at the invitation of Nephi and his brothers to join them on their journey.  The two families eventually intermarry. Ishmael later dies in the wilderness.