In this paper we describe challenges and lessons learned from developing a mobile touch screen based assistive tool for people with dementia. We focus not on features of the tool but the general participatory design process that was applied. Insights presented are gained from interviews, focus groups and observations. We found that projecting problems on imaginary characters and using simple games specifically customized for people with dementia ease the process of eliciting user needs and realistic prototypes allow design evaluations with people with dementia early on.