The magnetic properties of the Fe-based nanocrystalline alloys are determined mainly by their grain sizes, and the mechanism of the variation of grain size with annealing temperature is an important issue in the study of nanocrystalline alloys. In this paper, the relationships between grain size and annealing temperature for these alloys within the primary crystallization temperature (Tx1) and the secondary crystallization temperature (Tx2) for 1 h are investigated, and a corresponding model is proposed. The physical mechanism of the fact that the grain size first decreases and then increases with the increase of annealing temperature is explained by using this model. It is found that the grain size has a minimum value when these alloys are isochronally annealed at the temperature near 0.6 times that of the melting point. Theoretical analysis results are found to be in agreement with the experiments data within the investigated temperature range. This investigation provides a means to obtain the smallest grain size quickly.