The ubiquitous computing combined with the advance of sensor technology creates a scenario in which the integration of several computing resources is used to keep a set of services and features available to the user whenever necessary. A particular trend in this area is the activity based systems, i.e., systems that are aware of the activity played by the user. In these systems, inference engine is essential to recognize user’s actions and allow the systems to adapt its behavior according to user’s actions. Though, the development of this type of systems is not a trivial task as the high rate of information exchanged brings challenges related to privacy, performance and information management. In this work we propose an architecture for activity inference systems. To achieve this goal, we define a set of components that perform important roles in the inference process. Finally, to show the feasibility of this approach, we designed, implemented and evaluated a system based on the proposed architecture.