Lightning Network - Custodial?

Excuse my ignorance. I'm trying to more deeply understand the scaling solutions proposed by the various crypto projects.

My superficial understanding of the LN is that you and the party you are trying to pay will sign transactions with your private keys, but not broadcast them to the main chain. You would batch as many transactions as you would like, and then finally close the channel by committing the net transaction to the blockchain.

If this is the case, how is LN custodial? Suppose I open a lightning channel with my local coffee shop and pay for a coffee by signing a LN transaction. The coffee shop can rest assured that the funds will be released, and I can close the channel anytime I like and the unspent funds are returned to me. Where does the trust come in?

This seems reasonable enough to me. Can somebody explain why LN is a bad way to scale? Thanks for the help, r/btc.