I think the reasoning would be that a nervous system to convey pain would be complex and in a smaller body there’s less room to have complex systems. You can only fit so much into a package. Another thing, not necessarily for lobsters since they are so long lived, is smaller creatures also tend to have shorter lifespans so a system to convey pain wouldn’t be as helpful if you only live for like a year instead of 100 years. It might help you survive another month or two but would it be worth all the resources and space? Who knows.

Can crickets and grasshoppers feel pain? Would pain help them live significantly longer and produce more offspring than grass hoppers who could not feel pain?

That’s all that really matters in the end, if pain helps them breed more. If it doesn’t then that nervous system would never develop or eventually be bred out of the gene pool.