SpoilerShow
The only way to get sent back is if you made a mis-step. Sounds like you are doing the first half correctly, so you don't get sent back at that point. But you must do the second half entirely the same, a full complete accurate repetition, starting with the first two steps (so that, coupled with the last two steps of the first half, you make four steps forward there). The final step of the pattern lands you in the small hallway, THEN you can go back and get the key