I suggest to hardware implement auto-disabling of the audio amplifier when not playing any audio.
Some power savings could be achieved and the device would be prevented from producing unexpected noises.

How:
The audio amplifier chip has an "enable" pin.
I think it would be a good idea to use a low pass filter fed from the audio output so it would disable the amplifier after the "dac" was set to 0 for long enough.
This way, it would not be necessary to allocate one of the ESP32 outputs to control the amplifier chip.