yes, it matters incredibly. Using preprocessor directives to avoid having code compiled will allow you to write code in a file that can't be compiled on some systems but is compilable on others.
Consider any application that's written to be cross-platform compatible. It's obvious that the OS API is different for Windows than it is on Linux, but if I wanted to write an application that can be compiled on both machines, I'd use the #if/#else preprocessor directives to allow this.
You also might want to conditionally execute other preprocessor directives.
Download pretty much ANY open-source C/C++ cross-platform application and look through any of the source/header files. You're bound to see lots of #ifdef, #if defined, #elif, and #else statements.