A Framework for Dynamically Updating Building Information Models using Mobile Robots and Computer Vision
Project Team
Kincho Law, Max Ferguson
Overview
Automation in construction and facility management could significantly improve efficiency and productivity throughout the lifecycle of modern facilities. However, for robots and autonomous systems to operate effectively in dynamic and unstructured environments such as construction sites, they must be able to infer or obtain a semantic model of the environment.
This project investigates how information from mobile robots can be used to dynamically update an existing model of the building environment with semantic-rich information. Previous efforts in Civil Engineering and Robotics have focused on developing static maps of the environment, often assuming that objects are unable to be moved. However, in automated construction and robotics, it is important to keep models current. In this project, we develop a new set of algorithms to detect and localize worksite objects using image frames from an RGB-D camera. We then use a mobile robot to scan buildings and use our object detection algorithms to add new objects to the corresponding building information model.
Research Motivation
Construction is one of the largest industries and contributes to about 10 % of the gross national product (GNP) in industrialized countries. Automation in construction provides a potential solution for increased productivity and safety in construction. However, for these automated systems to work safely and efficiently, they must have access to, or be capable of generating, a semantic model of the facility. We believe that the construction and facility management industry would benefit from a system that could:
- Automatically detect new objects in a building worksite, and add those objects to a building information model
- Infer information about objects on the site, such as their size, color, material and damage condition
- Be deployed on a mobile robot or drone for autonomous data collection
Creating and maintaining a dynamic model of the building environment still remains a difficult challenge in computer science, robotics and civil engineering. Recent advances in Lidar, depth-sensing [1], visual odometry [2], and related technologies [3,4] have made it possible to obtain a static representation of large-scale spaces with relative ease. In most cases, this static representation exists as a point cloud or polygon mesh. However, we have not yet seen any comprehensive examples where point cloud information from a mobile robot is parsed on a frame-by-frame basis and used to automatically update a large-scale building information model.
Research Objectives
The following milestones outline the main objectives of the research:
- Develop a computer vision algorithm to identify common worksite objects
- Develop a framework for real-time building information modelling
- Validate the proposed framework using laboratory and facility-scale tests
Research Outcome
2D-3D Object Detection System
We developed a system for identifying common objects in a worksite and automatically adding them to an existing geometric model of the environment. The system is composed of two components: A 2D-3D object detection network designed to detect and localize worksite objects, and a multi-object Kalman filter tracking system used to filter false-positive detections. The proposed system was validated using data collected with a purpose-built mobile robot. The mobile robot captured images of the worksite with an RGB-D camera and used the proposed system to spawn newly placed objects in an existing model of the worksite environment. The annotated RGB-D images have been made publicly available to accelerate future research in this field [7][read paper].
Worksite Object Characterization
In object characterization, the goal is to identify common objects in a scene and extract rich semantic information about those objects. A new 2D-3D object detection algorithm was developed for the detection and characterization of common worksite objects. The proposed system has applications in automated surveying and data collection, especially in applications which leverage unmanned aerial vehicles or mobile robots. The proposed system was deployed on a mobile robot and used to detect newly placed objects in a worksite environment [read paper][8].
Future work
Future work will focus on the following objectives:
- Operationalize the computer vision technology, allowing it to be applied in real construction or facility management scenarios.
- Use the dynamic building information model along with extensive simulation to predict collisions, near-misses and safety issues, in real-time.
Awards
- First Place in the 2018 PEER Imagenet Challenge.
One of the key objectives in this project was to establish a system that could extract valuable information from images of a construction site or facility. The PEER Imagenet challenge consisted of 12 tasks, ranging from the detection of spalling and shear failure to the detection of full-building collapse. In each task, the computer vision algorithm was given an image of a building and had to predict the correct label. Our object characterization algorithm outperformed all other computer vision algorithms on this task.
Publications
- Ferguson, Max, and Kincho Law. "A 2D-3D Object Detection System for Updating Building Information Models with Mobile Robots." IEEE Winter Conference on Applications of Computer Vision (WACV). 2019. [link]
- Ferguson, Max, Seongwoon Jeong, and Kincho H. Law. "Worksite Object Characterization for Automatically Updating Building Information Models.", ASCE International Conference on Computing in Civil Engineering (i3CE). 2019. [link]
References
[1] P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox. RGB- D mapping: Using Kinect-style depth cameras for dense 3D modeling of indoor environments. The International Journal of Robotics Research, 31(5):647–663, 2012.
[2] D. Nister, O. Naroditsky, and J. Bergen. Visual odometry. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pages I–I, 2004.
[3] N. Engelhard, F. Endres, J. Hess, J. Sturm, and W. Burgard. Real-time 3D visual SLAM with a hand-held RGB-D camera. In The RGB-D Workshop on 3D Perception in Robotics at the European Robotics Forum, pages 1–15, 2011
[4] C. Kerl, J. Sturm, and D. Cremers. Dense visual SLAM for RGB-D cameras. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2100–2106, 2013.
[5] I. Armeni, S. Sax, A. R. Zamir, and S. Savarese. Joint 2D-3D-semantic data for indoor scene understanding. arXiv preprint arXiv:1702.01105, 2017.
[6] I. Armeni, O. Sener, A. R. Zamir, H. Jiang, I. Brilakis, M. Fischer, and S. Savarese. 3D semantic parsing of large- scale indoor spaces. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1534–1543, 2016.
[7] Ferguson, Max, and Kincho Law. "A 2D-3D Object Detection System for Updating Building Information Models with Mobile Robots." IEEE Winter Conference on Applications of Computer Vision (WACV). 2019.
[8] Ferguson, Max, Seongwoon Jeong, and Kincho H. Law. "Worksite Object Characterization for Automatically Updating Building Information Models.", ASCE International Conference on Computing in Civil Engineering (i3CE). 2019.
Original Research Proposal
Funding Year:
2019