By using the first-principles plane-wave pseudopotential method based on the density functional theory, substitution behavior and alloying effect of Nb in Ni3Al are studied. The analyses of the formation enthalpies,the cohesive energies and the densities of states of systems with different substitution behaviors show that Nb prefers to occupy the Al site. The site preference is determined primarily by the electronic structure, and the calculated result is in good agreement with the experimental result. In order to further study the site preference, the occupied sites are divided into being of relaxed distribution and unrelaxed distribution, which are two different distributions. The calculations of total energy, the formation enthalpy, the cohesive energy and the density of states of system indicate that Nb has a preference for unrelaxed distribution. In order to study the alloying effect of Nb in Ni3Al, the calculations of cohesive energy, density of states of system and electronic charge density indicate that Nb in Ni3Al could increase the stability of the alloy. The study also indicates that the limited solubility of Nb in Ni3Al lie between6.25at % and 12.5at %.