Inmy understanding, a cache line content is usually replaced (it is called cache lineeviction) only when there is no other space in cache to load necessary data. Eviction strategies can be different but obviously hot lines should not be evicted. Having in mind that with a N-way associative cache any address has only N places where it can be loaded, it does not seem a cache line will stay for long if not used. On the other hand, neighbour lines never conflict.
Back toyour example, I do not think the cache line holding pointers is immediately evicted after the pointers are dereferenced. As I said above, it will be done as its content gets "cold" and there is a need to place a "competing" cache line. The line holding the pointed-to memory is loaded as the memory is requested, that's right. And even if it competes for cache with pointers, it's not much likely the latter one is evicted as most probablyit is still considered hot.