Boil milk in a heavy saucepan. Add cheese a few tablespoons at a time stirring until each addition is melted before adding another.
Combine the brandy and the cornstarch into a smooth paste and add to the cheese mixture and stir until blended. Season with black pepper and nutmeg to taste.
Transfer the mixture to a fondue pot. Keep the heat source beneath the pot on just high enough to maintain the heat but do not bring to a boil. Serve with cubed dark brown or rye bread.