If you have more time in Pondicherry, you can also visit the Pondicherry Museum, Sri Aurobindo Handmade Paper Factory, and the Botanical Garden. For a beach day, you can visit Paradise Beach, Serenity Beach, or Auroville Beach.
If you have less time, you can skip the Goubert Market and White Town. Additionally, afternoon crowds at Promenade Beach can get a bit heavy, so you may want to visit in the morning or evening instead.