Test if dirt or mildew
Greetings! Two things can be happening. Mildew happens if the is a food source and wet conditions. You likely have both in the caulk. On the other hand, as elastomeric type caulks such as silicon, expand and contract with temperature it can embed dust and dirt inside the caulking and it becomes impossible to get out. In both cases, it isn't so much a failure of the caulk as it is environmental conditions.
A good test to see if it is dirt or mildew is to take a little bleach on a rag and wipe the area. Wait for a minute, it the stain visibly changes (disappears or turns from black to light brown), then it is mildew. If so, wipe it with bleach and let it dry thoroughly. You're all set. If it is dirt, then enjoy the 'road warrior, experienced' look. The only recourse if to remove the caulk and start over.
Hope that helps. Good camping!