The Galois group of Q acts on the homology of the complex moduli space of abelian varieties, or, equivalently, on the homology of symplectic groups Sp_{2g}(Z). (Here we take homology with finite or profinite coefficients.) In particular, the Galois group acts on the stable homology. Perhaps surprisingly, this action does not factor through  powers of the cyclotomic character. Rather, one sees nontrivial extensions between the torsion part and the torsion-free quotient. I will explain how to see this, and some consequences for families of abelian varieties. This is joint work with Tony Feng and Soren Galatius.