The simplest model of mirror sector dark matter maintains exact mirror symmetry , but has a baryon abundance \Omega _ { b ^ { \prime } } = \beta \Omega _ { b } and a suppressed temperature T ^ { \prime } = xT in the mirror sector ; hence it depends only on two parameters , \beta,x . For sufficiently small x , early cosmological observables may not constrain mirror baryons from constituting all of the dark matter despite their strong self-interactions , depending on the unknown details of structure formation in the hidden sector . Here we close this loophole by simulating mirror structure formation , mapping out the allowed regions of parameter space using cosmological and astronomical data . We find that the Milky Way disk surface density and bulge mass constrain \Omega _ { b ^ { \prime } } \lesssim 0.3 \Omega _ { b } at the highest T ^ { \prime } allowed by BBN and CMB ( T ^ { \prime } = 0.5 T ) , or \Omega _ { b ^ { \prime } } \lesssim 0.8 \Omega _ { b } at lower values of T ^ { \prime } . We also briefly discuss the realization of the necessary temperature asymmetry between the SM and the mirror sector in our model with unbroken mirror symmetry .