Yes, it IS steep walking up to Olive Gardens but it is quite do-able with a push chair if you take it slowly.  And of course there are always taxis waiting at the bottom by the supermarket if you're really desperate, and they're quite reasonable.  The tour operators DON'T leave you to walk up.  Our coach dropoff point and pickup was at the Pefanos office, but they laid on a minibus or taxis from there to the Olive Garden and  back again at the end of the holiday. The views and sunsets from the Olive Gardens are out of this world and well worth the climb.  Go for it, and just allow some funds for taxis in case you get desperate (2 or 3 euros to take you up the hill)!!