When signal passes through the A/D converter, the analogue voltage is converted to binary numbers. These binary numbers represent the amplitde of a waveform at any given time. Since there are an infinite number of analogue variations, the digital circuitry quantizes (moves to the nearest) to the bit depth you have chosen. The difference between the analogue signal and the quantized binary number is called quantization distortion. The more bits you have the less distortion because there are more levels of quantization and thus it is more accurate in its rounding. This effectively increases signal to noise ratio. The number of times a binary number is assigned per second is the sample rate (e.g. 44.1).
The difference in bit depth is most audible with low level signals like reverb tails etc and is also audible if the signal is truncated (extra bits are simply discarded) down to 16 bit instead of dithered down.
standard practice these days is to sample at 24 bit and dither (usually with some noise shaping) down to 16 bit for burning to cd.