Is there another bit that I need to set?
Pins A7 to A0 pin are shared with port A (PA).
If the value of the data direction register (DDR) of PA is 0 these pins cannot be set to address pins. Please check whether PA.DDR is set to 1.
For details, please refer to Port A (PA) in the I/O Ports chapter of the hardware manual.