At my day job I am currently working with a Spring boot Microservices stack that consumes and orchestrates many HTTP APIs. The data in most of these downstream APIs does not change a lot but it is read many times. Naturally this hints at using caching to improve performance. The code base already uses simple HashMap (not ConcurrentHashMap mind you) to do some local caching on per request basis within each micro-service. The next step would be to use proper caching technology following a common caching patterns for microservices. I decided to look at the first option* (i.e. Embedded Distributed Cache) and see how quickly I can implement one in my spare time (how hard can it be, I asked my self). I found an excellent resource from Piotr Minkowski to begin following his recepie and do some of my own cooking (And I think I found some ingredients of my own on the way).
Setting up local k8s tools
Basic Sprsing boot app and the magic of @Cacheable
Setting K8s deployment, service and probes
Hazelcast Embedded Distributed Cache
I have not done any extensive testing of my own when it comes to performance. that would be the next logical step to explore. But suffice to say that there are some pro and cons to this approach (as listed in hazelcast blog about caching patterns). I am a bit curious about how long it needs to take for the cluster to stabilize and how little is latency of this approach compared to say client-server in redis.
The data in embedded distributed approach is available to the application very easily. And it's colocated with the application. This should be considered in your design.
Overall I am very pleased to have tried this technology my self. I plan to continue hacking around in this area.