insights! episode #32: successful managed service in e-commerce

Joubin Rahimi:
Great to have you back for a new episode of insights! My name is Joubin Rahimi and today's topic is managed services in e-commerce. In the studio today is Marc Achsnich. Hello, Marc. We invited Marc today because he's been a specialist in this topic here at our company for several years. And Marc has a really interesting background, because he doesn't come from an operations or classic hosting environment, but actually learned software development from the ground up, as well as the topic of "how do I generate services at a commercial level". And that, combined with the knowledge we have here, forms an offering we call managed commerce. And today we don't want to introduce that to you, but rather the topic of "how do I do good managed services?" What's important there? So the first question to you, Marc, is what do you do all day to make it a good service?

Marc Achsnich:
These are actually different things. On one hand, we make sure the individual shop systems run stably. That includes making sure the shopping baskets work and the surrounding systems are connected. On the other hand, we're constantly trying to develop our e-commerce managed services further. So we see this more or less as a product and consider what individual functionalities might still be missing. What services can still be offered, and how can we ultimately make the shop systems faster, more stable and more secure?

Joubin Rahimi:
In the end, there are a few topics you just mentioned when I think of managed services or operating an e-commerce platform. How can I make this secure? Stable, fast, monitor, make sure orders go through? Then in the end you also mentioned partners. Straight away five, six topics your team works on, you have similar issues and go through this too. What are two, three things from the past where you'd say, in managed services, those were really important to have under control? Where you'd say that's a game changer for customers if they can do that?

Marc Achsnich:
Ultimately, one of the points is catching peaks. There's always a baseline in the shop, and the shop needs to run stably there. But customers of course also run many campaigns, and then there are unexpected surges in visitors, and a system needs to be able to scale and respond to that. It's particularly bad, of course, when visitors notice a campaign's running but the shop isn't there. And that's fatal, because these are large marketing costs being spent, and if the foundation isn't there, ultimately that's not good for the customer.

Joubin Rahimi:
Then you're basically handing out free colas for nothing. But how do you handle absorbing these peaks?

Marc Achsnich:
We, for example, set our system up on Kubernetes and have load balancers there, and have the system — taking Shopware as an example — set up across different legs, so we can scale the frontend as needed, so we can absorb load and check how many users are on the site now, how the load on the system is, and then scale the system to any size early on.

Joubin Rahimi:
So do you do that, or does system do it itself?

Marc Achsnich:
The system does that itself. So it checks it, there are its own status values that the system checks itself for the individual frontend applications, and if it finds that certain queries are taking too long, new frontend applications are simply added, and that within one to three minutes.

Joubin Rahimi:
That actually leads me straight to the next point, where you say scaling is important. But you have to identify when you need to scale. So you need to monitor it. And in that respect, do you monitor the speed parameters, how fast is the response time? But what else do you monitor? What's important in that context?

Marc Achsnich:
In fact, we have something we call 360-degree monitoring, meaning we really try to put the system through its paces. We start with the basic hardware-level information, be it how much CPU is available, what the memory utilisation is like, how much disk space I actually have left. On the other hand, it goes as far as checking how much revenue the shop is currently generating. How many orders have come in over the last 15 minutes? Are there any anomalies compared to the previous week, for example?

Joubin Rahimi:
Why is that important?

Marc Achsnich:
It might be, for example, that Saturday afternoon normally sees a lot of revenue compared to a Friday morning. So, if no order comes in for over 15 minutes on a Friday morning, that's initially a normal state. But if on Saturday evening, which is normally peak time, there's no revenue for a certain period, and in the previous week – that's a comparison you could use – you typically had a certain revenue, that's an indication there's a problem. And the problem could lie at the interface to the ERP, it could be that an error was actually introduced in the shop, whether through a connection to surrounding systems or because a bug was introduced through development. And those are the points where we can act early and also check whether there are any issues with the system. Because if no revenue is being made, that's bad for the client and, ultimately, for us too.

Joubin Rahimi:
Actually you have several layers of maintenance, of monitoring. I'll describe it in my own words and you can then say, wrong, that's missing. The first is close to the system, that's the no-brainer. The second is monitoring of surrounding systems, you said. Are they there or not there? And the third, I'd say a symptom monitoring: If no order can be placed, whatever the reason, and none of the other systems trigger, there's still a problem, even if it wasn't monitored. So checking, does this fit from a business perspective? And if not, that a human then looks at it? Is that described correctly?

Marc Achsnich:
Exactly. We specifically call this functional monitoring, meaning we check from the user's perspective, is the shop also functionally present? Do the goods work, does checkout work, does login, logout, product search work? Are the items even available? It could ultimately be the case that there are problems and the shop still exists. That means simply no revenue comes in, even though the shop ultimately exists.

Joubin Rahimi:
Or if availability simply isn't provided, you can't buy anything, at some shops, not all. Okay. You've mentioned backend system integration a few times now. Shop doesn't stand alone after all. How do you handle that?

Marc Achsnich:
That depends on how much information there is or how close the systems are. Sometimes a basic check is enough, to see whether the VPN tunnel between us and the surrounding system exists. Sometimes, though, it's also possible to query the surrounding system directly, such as the ERP. Is our order, which exists on our end, also present in the surrounding system after all? The simplest thing you can check is basically whether the system is reachable through the tunnel via ping?

Joubin Rahimi:
Okay, that's the basis then, and the other things are the functional tests and monitoring on top of that. If I monitor something, an alert comes out, or should come out. How do you handle that? Do you get alerts every five minutes? Or none at all, because everything's running smoothly?

Marc Achsnich:
Exactly. The answer is: hopefully not. There are also two to three different points of view on this. We have proactive monitoring, meaning we have large monitors in the individual offices, where we have the individual customers' spots, allowing us to try to directly identify anomalies even before the system has detected them. The system can detect anomalies, and if it does, there are three ways we get notified. That means, on one hand we have a channel where we log production incidents and warnings. We can be informed by e-mail. And the last point is that the system calls us directly. That means, depending on the customer and the SLA, it's also possible that we simply get a call from the system itself at night. Saying, here the checkout mechanism isn't working, the shop is down. And then we act and fix the problem.

Joubin Rahimi:
And this proactive monitoring happens when people simply take a look.

Marc Achsnich:
Exactly. Proactive means we try to fix the system before the problem actually becomes relevant. That means we analyse the history. We might already see that peaks can occur. We see that the hard drive is filling up. And we then try to alert in such a way that the problem is resolved before it becomes critical.

Joubin Rahimi:
You said calls come in at night too. There's for example Follow the Sun, where different teams around the world take these on. I know you've also tried putting the core team or first-level support on it. How do you do it? How did you set it up, and why did you set it up that way?

Marc Achsnich:
We have actually found that Follow the Sun does not work for our system. We have already automated this quite a lot and the very simple errors can be resolved by the system itself. This means that if manual intervention is necessary, it is not about working through a playbook, but rather you need to know how the system works. And that is why we developed a mechanism made up of a core team that looks after things during core working hours together with our service core team. And during on-call hours we have a team of our senior staff specialists who, on the one hand, know the application side and, on the other hand, also understand the administrative side. And they work weekend shifts in pairs, sharing the on-call hours between an active and a passive period. And that is how we manage to achieve response times of up to 20 minutes.

Joubin Rahimi:
That's really fast compared to other situations one knows. There's one point I wanted to get to, which I found interesting, because you said the best depth specialists are people who also know the application. How important is that, knowing the applications?

Marc Achsnich:
The problem ultimately is, once the system stops working, you have to keep a cool head to understand what exactly the problem is now. For example, the database no longer accepts requests. Why is that? You could now restart the database. After that it stops accepting requests again. Then you could restart it again and you'd have the same problem again. Which means you have to know exactly where you can check which requests are currently going to the database. What's necessary or typical for a shop, for example? HCL reacts completely differently. And so you really have to understand these individual systems and know exactly how they work, where the individual imports are listed and where you can now see what individual processes are currently running. So that you can then ultimately fix them.

Joubin Rahimi:
Thanks. Such an incident can also come from outside, I'm getting at data security now. Someone launches a DDoS attack or tries to hack. There have even been letters saying, we'll hack the system, please give us some bitcoins and we won't do it. We've never done that, of course. But how do you handle this topic of security and cyber-attack protection?

Marc Achsnich:
That's a topic made up of several components. For instance, you can put a Web Application Firewall right at the very beginning, before it even reaches our website. This would then filter all incoming requests and, per se, filter out DDoS attacks for example, hacker attacks, SQL injection. There's an institute for companies that deal with OWASP. Ultimately those are the top 10 attack scenarios, which are classified and prioritised, and that ultimately reflects various attack patterns. And these need to be eliminated in various ways. One point is a WAF that you can put in front of the application. We also work with security components, so that we in turn check our code as well, meaning before every check-in we check whether we're opening any security gaps anywhere, before the WAF would ultimately determine that it needs to protect somewhere. It's not enough to install one component and believe you're completely protected; it's about the combination of many. For example, we also always took that into account when choosing our underlying hosting provider. That means we've now migrated our Shopware Cloud to OVHcloud, because there you get an offering that is on the one hand data-sovereign, and on the other hand it's one of the few that offers its own dedicated Data Protection department. That means if there are hackers attacking some shop system somewhere that's hosted within OVHcloud, that department would identify it and block it directly for our network.

Joubin Rahimi:
Great. You've now taken three out of several components. Let's make that another additional session, I'd suggest. If you have questions, remarks, comments, just post them below or send us a direct message. We're happy to give you the answers here. Thanks, Marc!

Marc Achsnich:
Thank you.

Joubin Rahimi:
See you soon then.