Pose Estimation and Time Series Classification Methods for Efficient Video-Based Exercise Classification

The conventional approach to human exercise assessment is based on visual analysis where participants perform specific movements under the supervision of physiotherapists or trained coaches. However, this approach is inefficient and time-consuming at a large scale, and its outcomes are subjective in nature. Approaches based on sensors such as Inertial Measurement Units (IMU) have gained popularity with the advent of new technologies. These work by fitting sensors on different parts of the human body and using the output from these sensors for movement analysis and feedback. Despite being efficient, these approaches suffer from some key challenges such as the requirement for synchronization of multiple sensors, potential hindering of movement and discomfort when sensors are used over extended time periods. To overcome these challenges, using videos for data collection seems to be a promising approach as videos can be easily recorded using smartphones. In addition, there has been a notable increase in the popularity of video-based technologies for capturing and analyzing human movement in recent years. Hence, this thesis focuses on utilizing videos for the task of human exercise classification. The objective of video-based human exercise classification is to classify different executions of an exercise recorded using smartphones or dedicated cameras. As a result, this will help physiotherapists and sports coaches by delivering timely feedback to the participants to prevent injuries and maximize performance. Existing solutions to video classification face key challenges such as the requirement for large amounts of storage and computation, robustness to video quality and the need for near real-time feedback. Consequently, these challenges make these solutions inefficient for practical application. On the other hand, existing commercial products rely on rule-based approaches that fail to capture the complete dynamics of exercise execution. Hence, to overcome such limitations, in this thesis, we propose a novel and lightweight methodology for efficiently classifying exercise execution from videos. To achieve this, we first focus on a feasibility study in which we combine computer vision and machine learning techniques for video-based exercise classification. The study's outcome is the development of a new methodology that has been empirically demonstrated to achieve desirable performance according to domain experts, outperforming existing state-of-the-art video classification methods while also minimizing memory and storage requirements. In the second part, we examine the speed and robustness of the proposed methodology. Accordingly, we evaluate the robustness of this methodology against common sources of noise and empirically demonstrate its consistent performance up to a certain threshold of noise. In the third part, we compare video-based classification with sensor-based classification which is the more common approach. Experimental evaluation demonstrates that using a single video achieves significantly higher accuracy as compared to a single sensor-based approach. The main goal of this work is to pave the way for using videos for the task of human exercise classification. We believe that the results and findings presented will provide the necessary groundwork and guide researchers in bringing video-based human exercise classification closer to practice. All code and datasets created for publications connected to this thesis are publicly available.