When the particle size of silica gel is in large range, we always think that it is not uniform particle size. Sometimes, if all of them are large size, the solvent will pass quickly without resistance. But if there are small size, the solvent will pass slowly with high resistance.

But if there are both large and small size, the solvent will pass through the silica in a way with lowest resistance instead of pass through the column directly. 

This unsteady flow will affect the separation effect, and make the retention time different.