The annual Jazz Festival in Piedmont Park will be going on Memorial Day weekend. Luis and I went to this last year (on accident, since he was planning to propose in Piedmont Park haha). It is free! There are food stands, art market, and music. It will be going on May 24th-26th.
Red Top Mountain is approximately 30 minutes from Marietta Square, and has several lake-front hiking trails ranging from 1 mile to 5 miles. We have come here several times with Chase and Cara, and it is a great way to spend an afternoon!
Kennesaw Mountain is about 10 minutes from the square. It is the site of a Civil War battle. In additional to historic cannons, earthen fortifications, trenches, and monuments, Kennesaw mountain also has great hiking trails ranging from 1 mile to 10 miles, with views of the Atlanta skyline from the top.