Flexible pipeline architecture
Flexible pipeline architecture improves item processing efficiency at toom
toom operates around 300 markets and an online shop with a cross-channel approach that integrates features such as Click&Collect. Efficient processing of product master data is essential for a seamless shopping experience. The rapid growth of the platform highlighted the need for a customized software solution. Working with toom, we developed a flexible, event-driven pipeline architecture that accelerates and enhances data processing. This solution improves performance and refines the rendering of complex product variants, providing a solid foundation for future developments in the online store.
// Drastically reduced item data import times
// Modular architecture ensures long-term adaptability
// Customized software solution
// Kafka
// Microservices
// Product Information Management (PIM) system
// Data aggregation and transformation
// Modular structure
// System-wide testing
Flexible pipeline architecture improves item processing efficiency at toom
The hardware store chain toom has around 300 locations across Germany and an award-winning online shop. For years, it has implemented a cross-channel approach with features such as Click&Reserve and Click&Collect. The processing and enrichment of product master data is essential to provide customers with a consistent shopping experience.
Project background
The architecture of the Toom online store is based on a combination of custom-built microservices and secure third-party software. Standard software offers many advantages, especially in the early stages of development. It can be deployed quickly, is well tested and allows developers to focus on developing key functionalities. The processing of product master data at toom also follows this approach. It is regularly replicated from the Product Information Management (PIM) system to the shop software and exported to connected services for further processing.
The rapid growth of the platform placed new demands on the existing software. Additional features required adjustments to the data models in both the PIM and shop systems. At the same time, the volume of data being processed was increasing and numerous internal services required efficient access. A bespoke software solution for toom proved to be the best choice. This solution was designed and successfully implemented by experts from Qvest Digital AG, in close cooperation with Toom's developers.
When we began the Cross-Channel project in 2016, we were introduced to Qvest Digital AG through a recommendation from our sister company, REWE digital. Since then, our collaboration with Qvest Digital has been characterized by deep trust, commitment, and quality. The team led by Kai Ebenrett takes pleasure in exploring new paths, thinking 'out of the box,' and has accompanied us from the outset on the journey to becoming the best Cross-Channel DIY retailer in Germany. My colleagues and I look forward to the upcoming years of collaboration and the exciting new topics that lie ahead.
Our solution
The development utilizes a modern, event-driven pipeline architecture. Kafka, serving as the event broker, buffers each intermediate step, allowing consuming services to respond instantly to relevant changes. The pipeline is composed of four distinct layers, each managed by its own microservice:
- Import: Replicates data from the PIM system
- Transformation: Handles data aggregation and formatting
- Export: Transfers data to the shop system
- Post-processing/Validation: Monitors the processing and import of calculated data within the shop system.
As a positive outcome, the performance of importing item data has drastically improved. What once took up to 30 minutes for individual updates now completes in just a few minutes. By prioritizing flexibility and scalability in the architecture, we've established a strong foundation for the shop’s future development. Additionally, complex product variants were mapped for the first time, leading to a clearer and more user-friendly presentation.
Challenges and Opportunities
The transformation layer should map and extend the functional logic of the previously used third-party solution. However, this business logic was incompletely documented at the start of the project. Our agile approach allowed us to quickly identify and implement the business requirements. A modular structure and continuous refactoring helped to keep the architecture lean and maintainable.
The system-wide testing of the solution was challenging in terms of communication. Thanks to the commitment and cooperation of all internal and external teams, this was completed within two weeks, followed by a successful go-live with a subsequent hypercare phase.
Conclusion
The modular, event-driven architecture not only meets all stability and performance standards but also ensures long-term adaptability. This flexible, forward-thinking solution enables seamless future updates to the system architecture. By decoupling services, outdated components can be replaced with ease and efficiency. Additionally, the collaborative effort reinforces our client's position as a leading cross-channel online retailer in Germany.
Together with toom Baumarkt GmbH as our client, we implemented a complex IT project in an extremely agile and highly professional environment within a very short timeframe. Despite intense pressure on the project, the collaboration was very efficient and goal-oriented.
