Absolutely you should. The water filter helps to get a cleaner, stronger flavored brew from your grinds by removing the impurities in your water.
As the water runs through the permeable outer casing, the charcoal acts as a barrier between you and the impurities such as bacteria, chemicals and minerals by absorbing them and bonding to the charcoal pieces. When these impurities pass next to the carbon surfaces, they attach to them and are trapped.
The good thing about the charcoal water filters is that not only do they trap impurities but they also allow the good, natural minerals we get from water pass through.