• semantic
• map
• databse
• objects
• localization

Understanding the world around us is crucial for making decision in every days action. This concept has been applied in robotics in the last decade to create complex behaviors. Convolutional neural networks have been trained to detect objects and classify them providing a semantic information.
In this work, we propose a framework for semantic instance-based classification, mapping and localization in a 3D environment.
Starting from RGBD images, a semantic point cloud associated to each detected object is computed. In the mapping phase, all points are integrated probabilistically in a octree-based map which includes occupancy, semantic and color information. Simultaneously, an online database is created to contain all data about the semantic instances in the environment. This allow to retrieve information from the database efficiently and use it for a variety of application. In particular, the database is used to estimate the 2D pose of the system using only a RGBD sensor camera and the semantic information in the scene. To this purpose an Augmented Monte Carlo Localization algorithm is implemented using two different measurement models to extract semantic features.
Differently from state-of-the-art approaches which classify unrelated pixel in the scene and thus build a dense map, we exploit the knowledge of the instance segmentation to map and store only the objects. This allows us to have a perception of how many objects there are in the environment, their semantic, and thus their spatial relationship in the map. Results show that we achieve accurate localization with odometry and semantic feature extraction. Moreover, the proposed framework, through a set of API for its database, opens up to a large variety of user-defined complex tasks.