Structuring real-world code using monadic abstractions
In this article I present a simple relatively boilerplate-free approach to structuring Scala code using monadic abstractions. I use cats but it can be done using any other library, like Scalaz.
Presented approach is nothing new but just well-known patterns used in a cohesive way. In a sense it tries to compete with free monads or more modern Eff monad without going crazy too much. Those techniques are much more complicated and consequently more powerful, but they require a lot of boilerplate (even when using some libraries born to reduce it, like Freek or Liberator for free monads) and refactoring.
On the other hand, monadic abstractions approach in a sense is a less radical step forward to pure functional programming. It's much simpler and can be more easily applied to existing code that uses
scala.concurrent.Futures directly, without re-writing it from scratch.
Step by step guide by example
So, let's pretend we're writing a microservice that uses Scala futures for asynchronicity, Slick 3 for relational database access, and something like
akka-http to make REST calls to other microservices. First thing we do is...
Let's say we have a service layer and a repository layer. The first trick is to abstract away from concrete "monad" used as a methods result types wrapper, e.g.
DBIO, in interfaces:
Currently dominant style is to use
M in service trait and
M in repository traits. There are two downsides with such approach:
- using futures in repositories would force to evaluate your
DBIOactions too early (before leaving repository layer) effectively disallowing a transaction to span across multiple repository calls;
Min repository interfaces would leak repository layer implementation details to upper layers, adding `Slick* stuff to services classpath.
So, by abstracting it out, we want to be able to run cross-repository calls transactions and fix the implementation details leak. And we don't really want to have Slick classes in service layer auto-complete!
We use higher order type parameter
M, saying that we return a "container" of something, but we're not explicit about its type. It can be Scala
Eval, or even
Next we implement repositories by using
DBIO to substitute abstract
Implementation details aren't relevant here and therefore omitted for brevity. More interesting is how we implement the service.
Notice that service implementation should not depend on repository implementation but only on interfaces. We can say that each layer (e.g. repository, service, etc) has at least two parts, and consequently build artifacts: the
api part and one or more
impl parts. So each layer above only depends on
api artifact of a layer below. Explicitly:
apidoesn't depend on repository artifacts, cause it would make them available to layers above - something we what to disallow;
impldepends on repository
impl!) so that repository implementation could be swapped without changing service code (ideally);
- there is an app layer on the very top, where all the wiring takes place, it depends on both service and repository layers
apis transitively) and provides a concrete repository layer implementation to the service layer.
But how the service code looks then? Here comes the most interesting part:
Let's iterate it one thing at a time.
Notice that service code is still generic in the sense that it's detached from a concrete monad implementation, we're still using
M[_]. And it's a good sign!
DbEffect type parameter
Additionally, our service implementation declares a type parameter to abstract over a concrete "monad" (or "effect", or "container") of the repository layer. We could call it just
D but I like the
DbEffect name as more self-explanatory here. We also parameterize our repositories dependencies with
DbEffect effectively saying that we depend on repositories that produce values within a generic container called
DbEffect - and that's all we know about them so far, but... stay tuned!
And yes - we're using constructor parameters for dependency injection.
We've also got a fancy looking
evalDb: DbEffect ~> M constructor parameter. It can be written more explicitly like this:
In cats' code it's just a type alias for
This thing is called a natural transformation and all it can do is to convert a value of one generic type
F[_] to another generic type
G[_]. And that's exactly what we need to "run" our underlying
DbEffects: whatever monad service code uses for its results, we want to transform repository effects into it. For
DBIO actions, when we pass
evalDb implementation to our service in the app layer, we are going to execute them transactionally and get back futures (if we decide to use them as service's
There is one aspect of
M that gives compile-time guarantee of not having long-running transactions (what is probably a good idea). If you decide to do something like making REST call when transaction is running, you'll get a result of type
DbEffect[M[DbEffect[T]]]. And you won't be able to escape out of it. Strictly speaking, there is a typeclass called
Traverse that brings the ability to "traverse" one context with another, i.e. go from
G[F[T]], and since we can't have it for
M (at least when
M is some asynchronous boundary like
Future), - we cannot traverse it!
Now to the...
To recap, we require the following implicits when creating the service:
In case you know what is a
Monad (e.g. from my previous article) but never saw the
MonadError, you can think about it like a
Monad with built-in error handling. You can use standard Scala
Future methods intuition, like
Future.recoverWith. They allow to create a failed value and to recover from a failure. Same for the
MonadError typeclass but in much more abstract purely functional setting.
Why we require
DbEffect? It's just a rule of least power: I'm not going to use error handling for
DbEffect values in the method implementation so I just stick with the
Monad instance for
DbEffect but it could definitely be
MonadError[DbEffect, Throwable] as well - no trickery here!
So, by having this implicit parameters we declare that:
Mis a monad (with error handling) and consequently we can
recovervalues of type
M- same way as one would use
Futures in for comprehension;
DbEffectis a monad, so we can chain
DbEffectvalues as well, before transforming them to
evalDbnatural transformation (you can think about it like about transaction).
Wiring it all together
Before actually implementing our service method, let's see how the components can be wired in the top app level. Here are some common Slick helpers to be used:
I won't go into much detail here, this is something similar to what many teams come up with when using Slick. The most important thing is to be able to define
~> instance that can evaluate
DBIO values. Notice that we're using same "abstracting interfaces" approach for shared common code as for the components above.
Now, wiring can look similar to this:
For simplicity I'm not going to define web layer, but it might use
akka-http and require services dependencies as
securityService: SecurityService[Future]-like constructor parameters (or even
securityService: SecurityService[M] if you decide to generify your controllers/resources/whatever-you-call-it as well, though there's less benefit from doing that in comparison to service code).
But what about monad instances?
An astute reader might notice that something is missing when creating the
SecurityServiceGenericImpl above - namely the implicit
The first is defined in cats and can be put in scope by adding
import cats.implicits._ import.
As for the second one, there is a project on Github called slick-cats that defines
MonadError and some other instances for
DBIO. They can be brought in scope by introducing a dependency on
slick-cats and adding
import com.rms.miu.slickcats.DBIOInstances._ import statement.
Finally we're ready to implement our example service method.
Service method implementation
Method logic is following:
- get user's password last changed date;
- independently, get user's password validity period setting;
- if password is expired, call some other microservice to send a notification email to the user;
- if an error occurs during the notification sending, recover error to a specific
Even in this simplified example we have multiple things to consider, namely:
- we want to make first two calls independently on each other, i.e. no
- execute both calls in a single transaction (it might be more important in more real-world scenario than here);
- return a
ErrorWhileSendingADT value if an error occurs during notification sending (error handling);
- though we imply that repository calls cannot fail in our example, we might want to provide
DbEffectand handle DB errors gracefully (not covered here).
Here is the implementation split to several methods:
Steps 1 and 2 are implemented in
isPasswordExpired method. Notice that it doesn't leave
DbEffect monad yet, effectively causing both repository calls to execute in a single transaction, when run later by
evalDb. This method uses applicative builder syntax to leverage the fact that
Monad is also an
Applicative functor. In a nutshell,
Applicative is a less powerful abstraction that allows to, in this case, apply a function
(A, B) => C to
F[B] to get
F[C]. First, we use the fact that
DbEffect is an
Applicative, then the fact that
Option is also an
Applicative (typeclass instance for
Applicative[Option] provided by cats).
The main difference between
Applicative is that the former allows for computations chaining making one computation to depend on another, while the latter doesn't allow chaining and considers computations simultaneously, even with ability to run them in parallel when looking at how
Applicative[Future] instance works. See cats documentation for more information.
sendNotificationIfPasswordExpired method we
isPasswordExpired method result and, since it has
DbEffect[Option[Boolean]] type, we can apply
OptionT monad transformer to it. Monad transformers (hence the
OptionT) allow to stack one monad on top of another and work with them without nesting. Again, google for cats monads transformers if needed. In
semiflatMap method lambda we're already in
M "domain" and we call
sendExpiredPasswordEmail method, mapping it to
NotificationSent ADT on success and recovering to
ErrorWhileSending ADT on an error. In the end of the method, to get out of
OptionT transformer, we call
getOrElse method and return
UserNotFound ADT if we were unable to get user data out of our repositories (any of two calls).
sendExpiredPasswordEmail method implementation is supposed to make email sender microservice call, here we pretend it does it successfully and use
pure monadic operation to wrap
Unit value into monadic context.
As you can see, we are very abstract about what monads we're using in the service implementation. This allows to decouple application layers and change our monads without touching the service code. If one day you decide to go for Monix'
Task instead of Scala
Future, you won't need to change the service implementation at all. Just provide the required
MonadError instances (and yes - Monix does have them!) and all' keep working.
Applications to unit testing
Last thing to touch upon is how this approach affects unit testing. Instead of dealing with asynchronous results in unit tests (even though ScalaTest has a decent support for them), you can instantiate the service by substituting
cats.Eval as your
Notice that cats have
MonadError instance for
Eval in the master branch at the moment of writing, but it's not released yet.
Using ScalaTest and Mockito, you can do something like this:
FunctionK.id[Eval] that's just a "higher order identity"(i.e. like Scala
identity but applied to
M[_]) to "evaluate"
DbEffect values. So, same
Eval monad for both service and repository layers in tests.
It's worth saying that this test reveals another problem with our implementation: invoking side-effective methods like
LocalDateTime.now() in the service code. Instead, some date/time provider component should be passed to the service constructor to make the code more purely functional and consequently testable.
Wrapping blocking APIs
To wrap blocking APIs - those that return immediate values and do IO during their computation - one can define a transformation from
Eval to whatever monad you use, e.g.
And then use it to transform blocking calls into
I hope you came away with something applicable to the real-world from the presented monadic approach to abstract and structure (are these two synonyms in a sense?) the code. You can imagine having more "effects" like
DbEffect in your code and that's completely fine if you have required natural transformations machinery for them in place.
The thing I like the most about this approach is that it is simple. If you looked at free monads or Eff, you know what I mean. It doesn't force you to rewrite your code from scratch and can be adapted iteratively: first you apply it to the commons code, like
DatabaseSupport in the above examples, then to app's bottom layers, then move higher to services, and so on.
Next logical step to investigate is how to abstract streaming - very important technique nowadays. It would be cumbersome to have akka-http, or Monix, or Reactive Streams classes along with
DbEffect. But that's the topic of another article.
You might also be interested in these articles...