In the first part of this thesis, we introduce two list detection schemes for the uplink scenario of multiuser multiple-input multiple-output (MUMIMO) systems. The proposed techniques employ a single lattice reduction (LR) transformation to modify the channel matrix between the users and the base station (BS). After the LR transformation, a reliable candidate for the transmitted signal vector, provided by successive interference cancellation (SIC) detection is obtained. In the proposed multi-branch lattice reduction SIC (MB-LR-SIC) detector, a fixed number of different orderings, generates a list of SIC detection candidates. The best candidate is chosen according to the maximum likelihood (ML) selection criterion. For the proposed variable list detection (VLD) scheme, an algorithm to decide if the current candidate has good quality or if it is necessary to further explore different orderings to improve the detection performance is employed. Simulation results indicate that the proposed schemes have a near-optimal performance while keeping its computational complexity well below that of the ML detector. An iterative detection and decoding (IDD) scheme based on the VLD algorithm is also developed, producing an excellent performance that approaches the single user (SU) scenario. In the second part of this thesis, a decoupled signal detection (DSD) technique which allows the separation of uplink signals, for each user class, at the base station (BS) for massive MIMO systems is proposed. The proposed DSD allows to implement the detection procedures proposed in the first part of this thesis in massive MIMO scenarios. A mathematical signal model for massive MIMO systems with centralized and distributed antennas in the future fifth generation (5G) heterogeneous cellular networks is also developed. A sum-rate analysis and a study of computational cost for DSD are also presented. Simulation results show excellent performance of the proposed DSD algorithm when combined with linear and SIC-based detectors.