So here's my attempt at why the homset functor \$$\hom : \mathcal{C}^\mathrm{op} \times \mathcal{C} \to \mathbf{Set}\$$ has to be contravariant in the first argument. Suppose we attempt to define the homset functor as \$$\hom : \mathcal{C} \times \mathcal{C} \to \mathbf{Set}\$$ such that any pair of objects \$$(c,c')\$$ is mapped to the set of morphisms \$$\mathcal{C}(c,c')\$$ between these objects. We have to define what happens the morphisms \$$(f,f') : (c,c') \to (d,d') \$$, and applying the homset functor to the morphism pair should give us a map sending morphisms in \$$\mathcal{C}(c,c')\$$ to morphisms in \$$\mathcal{C}(d,d')\$$.

Let \$$h \$$ be any element of \$$\mathcal{C}(c,c')\$$ and \$$h'\$$ an element of \$$\mathcal{C}(d,d')\$$ such that \$$\hom (f,f') (h)=h'\$$. I drew the following diagram (I actually drew a bunch of arrows for each homset, but that's hard to do in LaTeX):

$\begin{matrix} & & f' & & \\\\ & c' & \rightarrow & d' &\\\\ h & \uparrow & & \uparrow & h'\\\\ & c & \rightarrow & d &\\\\ & & f & & \\\\ \end{matrix}$

From this diagram, we get two compositions \$$f'h : c \to d'\$$ and \$$h'f : c \to d' \$$, so the morphism \$$(f,f')\$$ ends up sending the homset \$$\mathcal{C}(c,c')\$$ to \$$\mathcal{C}(c,d')\$$; not quite what we want. But it's easy to see that if the arrow \$$f\$$ was turned around, we could precompose \$$h\$$ with it.

In \$$\mathcal{C}^\mathrm{op}\$$, \$$f : d \to c\$$ as desired, so we get the following commutative diagram:

$\begin{matrix} & & f' & & \\\\ & c' & \rightarrow & d' &\\\\ h & \uparrow & & \uparrow & h'\\\\ & c & \leftarrow & d &\\\\ & & f & & \\\\ \end{matrix}$
where \$$h' = f'hf\$$, which looks a bit like conjugation in a group. So the functor \$$\hom\$$ sends morphisms in \$$\mathcal{C}^\mathrm{op} \times \mathcal{C} \$$ to functions between homsets defined by the previous formula.