If you prefer to visit the memorial site, go late afternoon; note it can be emotionally heavy — alternative: relax at your hotel or visit an easy family-friendly attraction.
Early short visit to watch sunrise over Angkor Wat — arrive before sunrise to avoid crowds and keep it brief for the toddler (bring a light blanket and insect repellent).
Aim to depart the hotel ~3.5 hours before your international flight when travelling with a small child — airport checks are straightforward but allow extra time.