"just in case" means "only because there is a very small chance" (that something may happen). It's quite common in a clause that begins with "but". (The "but" is by no means required, however.)