Statistical Analysis of KPIs: the Missing Link in the VDC Decision-making Process

Principal Investigators: Calvin Kam, Sadri Khalessi, Martin Fischer 

Research Staff: Devini Senaratna, Ivan Leung, Amelia Celoza, Wenli Yang

Engineering/Business Problem Scenario

Problem Scenario 1: Industries such as manufacturing and information technology, use metrics or KPIs more systematically and successfully. Esin et al’s (2009) study on the usage of KPIs in the apparel industry pointed out that as much as 80% had some form of quantifiable KPI/metric that was measured. It was inferred that though scorecards are available to track and measure BIM/VDC usage and performance, a methodology to analyze correlations and give effective feedback and recommendations is yet to be incorporated into the VDC Scorecard methodology. In other industries, methodologies that instantly analyze a project’s status and provide feedback are both available and effective (Zevenbergen, 2006) . 

Problem Scenario II: The present VDC Scorecard insights and recommendations are based on independent analysis of the VDC Scorecard KPIs. For example, if a KPI scored poorly with regards to VDC, feedback is given without considering the impact of that KPI on other KPIs. Initial statistical analysis show, complex correlations between KPIs. These can be used to provide useful feedback, such as forecasting performance, and benchmarking projects.

Project Proposal

Key Performance Indicators (KPIs) help Architecture, Engineering and Construction (AEC) project teams to make informed decisions. The Virtual Design and Construction (VDC) Scorecard research shows that only 40% of AEC firms use KPIs to evaluate VDC, as compared to over 80% in other industries such as textile. Other industries use KPIs efficiently by analyzing correlations and providing insights for decision making. This aspect is a ‘missing link’ in the AEC industry.

We will integrate statistical methods with KPIs to help AEC professionals in VDC decision-making. Building upon 108 cases from the VDC Scorecard results, we will conduct web-surveys to obtain VDC performance statistics to develop and enhance the KPIs. We will use statistical methods including Structural-equation Modeling, and Clustering to identify relationships between KPIs. We will present to CIFE members, as the final product of this SEED proposal, 3 statistical models for benchmarking, decision-prioritization, and prediction, to enhance VDC decision-making.

Test Cases

The VDC Scorecard research team carried out an extensive analysis using 146 unique VDC Scorecard evaluations, and over 200 other evaluations, to understand industry trends and key performance indicators related to VDC performance. This data set covers projects from 13 countries, 11 facility types, and all 7 stages of the construction process. Additional data will be collected from the existing VDC Scorecard cases. This includes both CIFE member organizations and other industry contacts. Results validation will be continuosly carried-out using two methods, a statistical data-based validation and an expert opinion based validation. For the purpose of validating the recommendations, CIFE members, stakeholders of the projects scored using the VDC Scorecard, students from CEE 112/212, and AEC experts will be required to express their opinions on the feasibility of the results, based on the characteristics of a few test-cases. 

Key Performance Indicators (KPI) Research Findings

Methodologies used toanalyse data include Spearman’s rank correlation test, t-contrast test of group means and Cluster analysis. The most important findings of this research are (See KPI_Workshop_March.pdf below for more details):

  • Tracking and monitoring targets: Planning objectives can be broadly classified into 7 major categories by their purposes: Cost performance; Schedule performance; Safety; Project delivery; Communication; VDC management; Facilities management. Previous workshop participants ranked the importance of setting objectives in each of the 7 categories. See diagram below to inconsistencies between industry requirements and practice. Results from the VDC Scorecard also indicate that AEC projects rarely have objectives in Safety or VDC Management.
  • Trends by year: Number of Cost, Communication, Schedule, and Project Delivery objectives established by AEC projects are increasing. But number of established Safety,VDC Management,  and Facility Management objectives did not show significant increase by year.
  • Owner involvement also pushes for more formalized VDC among project participants and stakeholders.

2014-2015 Research

  • Owner BIM phone survey: Study the habits, competence, purposes, and return of investment of BIM use from project owners' perspective.
  • 2014 McGraw Hill SmartMarket Report: CIFE researchers are collaborating with McGraw Hill Construction to co-publish a SmartMarket Report that evaluate the industry's trends in BIM use.
  • VDC Scorecard: Spring 2014 sees the completion of Version 9 of the VDC Scorecard with refined scoring metrics and questions. CIFE researchers will integrate results from statistical analysis performed in Spring 2014 to refine the VDC Scorecard. Version 10 will better evaluate a project's VDC use by incorporating more quantifiable metrics, especially in the realm of measuring return of investment for projects. Additional questions will be added to the VDC Scorecard to assist statistical analysis of VDC use in this year's research.

Presentations and Workshop Slides

Please click on the following links for further information on this project 

2013-03-CIFE-SEED-Proposal.pdf1.27 MB
2013-03-CIFE-SEED-PowerPoint.pps2.62 MB
KPI_Workshop_March.pdf1.91 MB

Last modified Fri, 8 Aug, 2014 at 1:30