The next generation of mobile infrastructure management systems will allow on-site infrastructure field workers, such as construction superintendents and bridge inspectors, to use mobile and wearable computers to interact with geo-referenced spatial models of the infrastructure and to automatically retrieve the necessary information in real time based on their location, orientation, and specific task context using Augmented Reality (AR) techniques. AR allows interaction with 3D virtual objects and other types of information superimposed over 3D real objects in real time. The augmentation can be realized by looking at the real world through a transparent head-mounted display equipped with sensors that accurately track head movements to register the virtual objects with the real objects in real time. In these systems, field workers will be able to access and update information related to their tasks in the field with minimum efforts spent on the interaction with the system, which results in increasing their efficiency and potentially, their safety. In this paper, we discuss a new AR interaction model that has been developed especially to suit the requirements of mobile infrastructure management systems. The model is based on automatic picking of the objects of interest and augmenting the 3D real scene with data extracted from a database. The algorithm of the model is explained and implemented in a prototype system for a building construction project.