The geometric structures, electronic properties, average binding energies, second-order energy differences and energy gaps of BnY (n=1-11) clusters are systematically studied using the density functional theory (DFT) TPSSh method with 6-311+G(d) basis set for B atoms and Lanl2dz relativistic effective core potential basis set for Y atom. It is found that with the size increasing, the lowest energy structures of BnY (n=1-11) clusters gradually evolve from planar shape to cubic structure. With the atoms of B increasing, the average binding energies of the ground state of BnY (n=1-11) clusters increase. The second-order energy differences and the energy gaps of the ground states of BnY (n=1-11) clusters show that B3Y, B5Y and B7Y clusters possess relatively high stabilities. The polarization and the first static hyperpolarizability studied show that the plane structures of B5Y, B4Y, B3Y and B6Y clusters have larger nonlinear optical properties.