The problem of portfolio optimization has always been a key concern for investors. This paper addresses a realistic portfolio optimization problem with floor, ceiling, and cardinality constraints. This problem is a mixed integer quadratic programming where traditional optimization methods fail to find the optimal solution, efficiently. The present paper develops a new hybrid approach based on an improved particle swarm optimization (PSO) and a modified simulated annealing (SA) to find the cardinality constrained efficient frontier. The proposed algorithm benefits from simple and easy characteristics of PSO with an adaptation of inertia weights and constriction factor. In addition, incorporating an SA procedure into IPSO helps escaping from local optima and improves the precision of convergence. Computational results on benchmark problems with up to 225 assets signify that our proposed algorithm exceeds not only the standard PSO but also the other heuristic algorithms previously presented to solve the cardinality constrained portfolio problem.