The challenge of improving construction safety performance is observed in many countries. Safety is considered by practitioners and researchers as an important topic in construction industry sites. Despite the findings of implementing safety programs, it is revealed that accidents and injuries are not perfectly reduced in construction projects. In the literature, authors tried to establish several frameworks and proposed methods to reach this objective by identifying the key factors affecting safety performance. The aim of this study is to present critical factors used in the implementation of safety programs and to explore their relationships using Interpretive Structural Modeling (ISM). Then, via ISM technique, the overall structure among factors was revealed. By using the Fuzzy MICMAC analysis, the factors were classified into four groups based on their driving power and dependence power. The results showed that “Safety Training” and “Management Commitment” have the most important impact on safety programs, but also it is very important to study the interactions among factors at different stages. This analysis offers key resources for practitioners and decision makers by analyzing the relationships between factors and its driving or dependence strength. These results shed lights on the effective development of measures to facilitate the implementation of safety programs in the construction sector.