Registers provide internal state storage for a device.
These may store a number, like the accumulator register in the CPU, or they may store other behaviour states, like the colour output of the PPU. The various cartridge mappers have registers that control banking, as well as other devices (e.g. IRQ timer).
A register might be entirely internal, or it might have a memory mapped address to provide external access via the CPU. Such registers are often referred to by this write address.