I'm sure other people tranistioning to agile have had this thought. Isn't agile supposed to make my job easier? Won't I have more time to golf and relax? And yet, agile seems to be harder, and require more work and effort, than doing things business as usual. Well, guess what? Executing great work at the highest level is hard! It's harder than business as usual. But it pays dividends. You delight your customers, you have a happy, satisfied team doing valuable work.
The moment the team stops doing work against stories its committed to, the magic leaves the room, quickly and dramatically. This is your fault as the product owner or business manager, not the teams. Business inputs will constantly distract you, customers will constantly derail you, personal issues will manifest in your team's lives. The world will never cease in its efforts to derail, and that's why you have the hardest job of all as the product owner. You're the glue holding the whole thing together, keeping the world at bay.
It actually reminds me a lot of skeet shooting. Shotguns are weird weapons, and skeet shooting is quite the challenge: you're trying to hit a moving target with an explosive burst of shot. You're not sitting there at the range, lying prone, taking as much time as you need to put the bead on the bullseye. You've got at most 2-3 seconds where the clay will be close enough to the target area where your shot will hit it with any force.
The magic of the shotgun is that you don't aim down a site. You line up your vision to the sites - you put your entire upper body and vision in alignment so that you can follow the target with your eye. You no longer aim. You see the target line up, you squeeze the trigger, and pow! it explodes.
This reminds me a lot of setting the goals for a sprint. It's all mental. You have to line everything up to hit the target before shouting "pull!" and if you don't, you're guaranteed not to hit this elusive target. Go down to your local clay shooting range and try it out, you'll find it weirdly identical to leading a software team. Take your whole team as a team-building excercise. There's something interesting that happens when you fire deadly weapons together safely.