You can only gain experience in versus online game modes. You can gain experience in Xbox Live private matches; but it's minimal at best and not a sustainable source of XP. You will gain experience for everything you do; but it mostly boils down to kills, wins and the consecutive match bonus. Without a doubt, this will take some time to accomplish.
Once you reach level 100, which requires you earn 3,851,750 experience, you'll have to visit the Statistics page to 're-up'. Once there, press (X) to re-up; the game will ask if you're sure, then press (A) to confirm. You'll revert back to level 1 and net yourself the achievement. If you're a Gears 3 veteran, you know all about re-upping; for those that don't.. think of it as a prestige.