​Lack of context as a writing style

I have noticed there is a spectrum of the level of information and context given to a reader. On one is explicit world building and character introductions, on the other hand there is total dearth of information: “Sally walked into Smith’s, it looked just the same way it always did.”

I understand there are many situations in which withholding information from a reader makes sense as a tool. A lazy example might be a crime thriller.

However, there are far too many examples (in my opinion) of authors referring to places, characters, objects, concepts, that simply could not be known to the reader, are not explained to the reader, and are often left unexplained for the entire book.

From my perspective as a reader, I find this immensely frustrating and sometimes even rude (if that’s possible). It is somewhat like joining a conversation at a party on a topic that everyone else is across but you: the polite thing would be for someone to loop you in.

When authors do this I find myself wondering whether I am just stupid: should I be inferring more than I am, and should that ability to infer be something that comes naturally. My next thought, however, is the opposite: how lazy of the author to have a full picture and not share it with me.

Does anyone else have the same, a similar, or perhaps an opposing view. This is something that has driven me mad for a while now.