Dragging to the trash is slow. Select a file and hit Cmd + Delete. To permanently delete (skip trash), use Cmd + Option + Delete.
