When you destroy an AudioSource, the AudioClip will not be unloaded from the memory. You can use UnloadAudioData to do so. This works only for AudioClips that are based on actual sound file assets.