You should think of pointers as being addresses of virtual memory: modern consumer operating systems and runtime environments place at least one layer of abstraction between physical memory and what you see as a pointer value.
As for your final statement, you cannot make that assumption, even in a virtual memory address space. Pointer arithmetic is only valid within blocks of contiguous memory such as arrays. And whilst it is permissible (in both C and C++) to assign a pointer to one point past an array (or scalar), the behaviour on deferencing such a pointer is undefined. Hypothesising about adjacency in physical memory in the context of C and C++ is pointless.