Smells have a very strong connection to both memories and emotions, but do you realize why that is? With most sensory information (taste, sound, touch, sight), the information you receive from these senses goes straight to the thalamus, but for smells it's different. This sensory information instead goes to the olfactory bulb which is a structure at the front of your brain. Through this, smells take a direct road to our body's limbic system which deals specifically with emotion and memory. As a result, we tend to associate both good and bad smells with specific moments in time.
I definitely agree with the idea that in general we have very strong associations of smell to memories even if we don't realize it. It was also mentioned that because of the fact that smell is one of the first senses we develop and is the most developed when we are younger, this is the time period we begin to associate smells as being "good" or "bad" and how that is very hard to change. A lot of us think something smells bad or good without even thinking of why and in reality it's probably associated with a memory or emotion.