Only keep your files as 24bit/wav. if the samples were originally sampled in 24 bit to begin with, or you recorded them yourself in 24bit, otherwise this is a waste of file space since your not going to be improving the quality of your audio in anyway by doing so.
Example: All your drum samples are in 16bit/44.1 > by converting to 24bit/xxx your only going to end up making larger sized files of the same quality.
Example: All your drum samples are in 24bit/xxx > by keeping your wav files at 24/xxx and not altering them (downsampling or compressing), your assuring that the samples will stay the highest quality possible.
But yes, to answer your question: sticking to wav files is a pretty safe bet since that particular file extension is still so widely applicable with various softwares.
*Keep in mind though, you dont have to keep the files only as wav's to have them stay around that quality level.....if you prefer aiff. thats perfectly fine too. Or maybe the software you use has its own compression format but maintains the same level of audio quality as the original wav? (perfect example of this would be Reason's Sample packs that you can make, or Rex files.) - well then thats perfectly fine too, just make sure to think ahead since by making a sample pack for Reason your basically nixing out any other opportunities to use those sounds in other host applications. My point - decide what your workflow demands and cater your file preferences around that instead since there are various options depending on how you work.
Long story long though, wav is a safe bet.