Flexible pipeline architecture

An event-driven pipeline architecture was implemented at toom to improve item processing efficiency. This custom solution boosted data flow performance, ensuring smoother operations. It future-proofed the retailer’s online store, accommodating growth and enhancing scalability to meet future business needs effectively.

Flexible pipeline architecture improves item processing efficiency at toom

The hardware store chain toom has around 300 locations across Germany and an award-winning online shop. For years, it has implemented a cross-channel approach with features such as Click&Reserve and Click&Collect. The processing and enrichment of product master data is essential to provide customers with a consistent shopping experience.

Project background

The architecture of the Toom online store is based on a combination of custom-built microservices and secure third-party software. Standard software offers many advantages, especially in the early stages of development. It can be deployed quickly, is well tested and allows developers to focus on developing key functionalities. The processing of product master data at toom also follows this approach. It is regularly replicated from the Product Information Management (PIM) system to the shop software and exported to connected services for further processing.

The rapid growth of the platform placed new demands on the existing software. Additional features required adjustments to the data models in both the PIM and shop systems. At the same time, the volume of data being processed was increasing and numerous internal services required efficient access. A bespoke software solution for toom proved to be the best choice. This solution was designed and successfully implemented by experts from Qvest Digital AG, in close cooperation with Toom's developers.

Our solution

The development utilizes a modern, event-driven pipeline architecture. Kafka, serving as the event broker, buffers each intermediate step, allowing consuming services to respond instantly to relevant changes. The pipeline is composed of four distinct layers, each managed by its own microservice:

1. Import: Replicates data from the PIM system
2. Transformation: Handles data aggregation and formatting
3. Export: Transfers data to the shop system
4. Post-processing/Validation: Monitors the processing and import of calculated data within the shop system.

As a positive outcome, the performance of importing item data has drastically improved. What once took up to 30 minutes for individual updates now completes in just a few minutes. By prioritizing flexibility and scalability in the architecture, we've established a strong foundation for the shop’s future development. Additionally, complex product variants were mapped for the first time, leading to a clearer and more user-friendly presentation.

Challenges and Opportunities

The transformation layer should map and extend the functional logic of the previously used third-party solution. However, this business logic was incompletely documented at the start of the project. Our agile approach allowed us to quickly identify and implement the business requirements. A modular structure and continuous refactoring helped to keep the architecture lean and maintainable.

The system-wide testing of the solution was challenging in terms of communication. Thanks to the commitment and cooperation of all internal and external teams, this was completed within two weeks, followed by a successful go-live with a subsequent hypercare phase.

Conclusion

The modular, event-driven architecture not only meets all stability and performance standards but also ensures long-term adaptability. This flexible, forward-thinking solution enables seamless future updates to the system architecture. By decoupling services, outdated components can be replaced with ease and efficiency. Additionally, the collaborative effort reinforces our client's position as a leading cross-channel online retailer in Germany.