If you happen to be in Puno when the festival of Candelaria is going on you should not miss it. It is impressive.
Party at every corner, beautiful costumes, very authentic people…fantastic.
The 4,000m above sea level is definitively notable. From Puno tours to the floating islands of Uros and more Islands are offered. I liked it, but it seemed very touristy, but it is worth it. The tours often can be combined with homestays. But you can find other islands where you can find more authentic people who will offer you to stay.