A new report calls on Massachusetts to reform its criminal justice system, saying prison spending is rising despite a decline in violent crime.