This is the "flehmen response"

You've seen it in pictures, movies, or perhaps in person - now you know what it's called.

The horse (or other ungulate or felid) curls the upper lip, exposing a region at the base of the nose called the vomeronasal organ, an olfactory chemoreceptor. I found this out today because Wiki featured a photo of a python and noted that the reason a snake's tongue is forked is so that when it touches it to the vomeronasal organ it can detect from which direction a smell is coming.

This, however, is apparently unrelated to the "talking" maneuvers of the television horse Mr. Ed. For that program "a loose piece of Nylon was inserted under Mr. Ed's lip which the horse attempted to remove on his trainers cue. Mr. Ed was so well trained that the insert would be ignored until the required cue."

Flehmen is a direct borrowing from the German flehmen, meaning "to curl the upper lip." It's not in my Random House dictionary (or my OED), or even in my decades-old Langensheidt's, so I can't offer any other etymologic relationships or related words.