When a passenger is already inside the elevator, and wants to travel in the wrong direction, the elevator should not ignore that passenger completely, but only ignore the passenger until it's done travelling in the current direction.
The current situation means that when you press the button for a lower floor inside the elevator, the button for that floor doesn't light up, causing confusion from the rider, who doesn't expect the elevator to not continue to his floor but stay put at a different floor.