We present theoretical and experimental results related to the problem of clustering a set of vectors (which can be interpreted as probability distributions) with the goal of minimizing a weighted impurity measure of the resulting partition. The problem of clustering while minimizing the weighted Gini impurity of the partition is shown to be NP-complete and APX-hard, via a connection with the geometrical k-means problem. We also analyze a family of algorithms for information-theoretic clustering that rely on the dominant (largest) component of the vectors to be clustered. These algorithms are shown to be very fast compared to the state of the art, while able to achieve comparable results in terms of the resulting partition s weighted entropy.