The rise of video content as the main media for communication has been creating massive volumes of video data every second. The ability of understanding this huge quantities of data automatically has become increasingly important, therefore better video understanding methods are needed. A crucial task to overall video understanding is the recognition and localisation in time of dierent actions. To address this problem, action segmentation must be achieved. Action segmentation consists of temporally segmenting a video by labeling each frame with a specific action. In this work, we propose a novel action segmentation method that requires no prior video analysis and no annotated data. Our method involves extracting spatio-temporal features from videos using a pre-trained deep network. Data is then transformed using a positional encoder, and finally a clustering algorithm is applied where each cluster presumably corresponds to a dierent single and distinguishable action. In experiments, we show that our method produces competitive results on the Breakfast and Inria Instructional Videos dataset benchmarks.