fix f = f (fix f)

First, f takes the result of fix f, and second, the return type of f must be the same as the return type of fix.

fix :: (a -> a) -> a

Research shows that fix is the Fixed point combinator (or Y-combinator). Apparently this allows the definition of anonymous recursive functions. So let’s try to make remainder into an anonymous function. A good way to start is to simply remove one of the arguments and see if we can return an anonymous function. Something tells me that a is the argument to remove, since b is used in the base case.

remanon b = (\f x -> if x < b then x
                     else f (x-b))

That's a good start. Note here that if we try to do something like:

(remanon 3) remanon

Haskell says "unification would give infinite type" - which is a good sign. This is what we would like to do with the help of fix. So now if we apply fix to this, we get:

fix (remanon b)
= (remanon b) (fix (remanon b))
= (\x -> if x < b then x
         else (fix (remanon b)) (x-b))

Which looks good. So we need to fix remanon and apply it to a.

remainder' a b = fix (remanon b) a

And that works.