The construction work environment remains one of the most hazardous worldwide. As a result of dangerous working environments, construction workers often face safety and health risks throughout the construction process. However, it is worth noting that an emphasis on construction safety has increased mainly in the aspect of safety performance, and efforts are required to not only monitor but also seek to improve safety performance. The main objective of this research is to assess safety performance on construction sites using safety activity analysis technique. A safety activity analysis framework as well as a safety activity analysis tool are developed to facilitate the collection and analysis of the safety activity data for continuous evaluation of safety performance. Additionally, a case study is carried out to implement the framework and tool on an active construction project. Activity observation is used to continuously collect data of safe behaviors and conditions, along with unsafe behaviors and conditions on construction sites for analysis and decision making. It is expected that the observation results can be used to set improvement targets and provide continuous feedback so that construction workers can adjust their performance accordingly. The findings of the case study show that the safety index of the construction site ranged from 37.0% to 62.8% with an average of 53.8%. Results obtained also indicate that the stage of a construction process can affect the safety performance of a project. It was recommended that well-informed decisions and far-reaching efforts have to be made to ensure the safety index is brought as close as possible to 100% by reducing the unsafe behaviors and conditions on the construction site.