Electrical Power Systems are cyber-physical systems responsible for the generation and transportation of energy from its generating source to the final customers. During this process many different activities must be conducted in order to keep quality of service and the system s safety and stability. One of these activities regards control of various equipment in order to keep the voltage level on each system bus between specified limits. This control, which is usually conducted by system s operators in real time and by automatic control equipment involves many different constraints and considerations that are hardly ever taken into account during the decision process. In order to mitigate this problem a smart agent capable of deciding which action is best in order to keep the voltages in adequate levels taking into account system s conditions is proposed. The proposed methodology consists on the Deep Reinforcement Learning technique along with three novel variations: windowed, ensemble and windowed ensemble Q-Learning, which consist on the division of the problem in training windows, the usage of multiple learning agents for the same process and on the combination of both these techniques. The variations are tested on academically consecrated test circuits and are capable of attaining expressive results when compared to the traditional Deep Reinforcement Learning approach which is used in other academic studies and also with the systems intrinsic control, keeping voltage under control along the day.