The Old Mission is beautiful and interesting in it's own right, and also provides a great backdrop for hanging out at the Rose Garden, located just across the street. One of our first dates was a picnic among the roses!
The pier is a great stopping point when walking around downtown; you can choose to go to an ice cream shop, Deep Sea Winery, Char West Burgers, or just walk to the end and watch the fisherman. The best time to go is sunrise and sunset - be sure to turn back to look at the mountains as well as the ocean!
More Mesa is a great place to go for a long meandering (read: get lost) walk along the bluffs. You can spot trail bikers, horse riders, and garden snakes; we often go here for evening runs! The beach is also beautiful and can be accessed from the bluffs... just be aware there's a nude section!