Sutton House Stories is an on going project through which we explore affective storytelling for situated learning in the cultural heritage context using smart glass Augmented Reality (AR). Our project looks at cultural heritage sites as opposed to museums that hold collections of objects. The reason for this is that heritage sites are physical places that incorporate both tangible and intangible heritage, they are the holders not only of the physical structure but also of the stories that happened in their grounds, and they offer an incredibly immersive experience without any digital technology. The stories that surround a historic site offer a rich playground to design affective narrative. We collaborate on the project with National Trust - Sutton House, a Tudor house in Hackney and an important place of history in central London. Through this research we are developing a framework for designing smart glass Augmented Reality experiences in the specific context.