Individually Developed BI is Overrated. The Truth is: 80% of BI Requirements in Commerce are Identical.
“Yeah, we're buying an ecommerce platform, ERP, CRM, checkout and marketing systems off the shelf - but our BI requirements are so highly individual that we need to build a solution ourselves that's tailored to our needs.” Yes, sure. You can handle your processes with the standard, but you must analyze in a highly specific way. Makes sense.
Don‘t get me wrong: In most retailing companies, there are individual premises for data analysis, sure - for example, company-specific metrics terminology or certain parts of the business model that require special analytical treatment. But the truth is: only a very tiny part of that actually justifies the development of an individual BI solution - at least 80% of the requirements can nevertheless be covered by a standard software solution, the remaining up to 20% are then either individually customized or extended within a standard solution (as with minubo) or supplemented by individual DIY components.
To better understand this, let's take a closer look at what "individualization" means in business intelligence.
Levels of Individualization in Business Intelligence
On the whole, there are four levels of BI, on which the individualization wishes of companies are focused:
1. Frontend Tools: Operational, Visualization and Analysis
Questions | Do my users have the tools they need? Can I customize the method of data visualization in reports, dashboards and so forth? Can I perform flexible analyses?
The Facts | Frontend tools that go beyond pure reporting, dashboarding, and classic analysis tools are a key requirement of modern BI, as this category includes operational tools such as customer or product segmentation or proactive alerting that enable most of the user base (namely the operational departments) to work data-driven. However, the fact is that by far the largest part of companies will never be able to develop such tools themselves for their DIY BI - which is why the cry for individualization has no validity here, at least not as an argument for a self-built BI. On the contrary, professional BI software vendors are much more likely to accommodate the desire for custom tools as part of their standard software. All other "classic" frontend tools (reporting, dashboarding, tools for ad-hoc analysis) can be easily purchased and set up individually as a component for every self-built BI solution (e.g., visualization tools such as Tableau or Qlik View, or analysis tools such as Excel) - which means that the desire for individualization in the field of visualization and analysis is usually perceived as a strong argument for a self-built BI. But in fact, any good purchased BI software is able to meet individual requirements in these areas, for example, with freely configurable dashboards, configurable reporting and flexible analysis tools based on raw data. If there are limitations here, good BI software will at least be able to deliver data from the consolidated data warehouse via outbound APIs to third-party systems that can complement the unfulfilled requirements - this "effort" would just be a fraction of the hassle that the development of a completely individual setup would be.
Conclusion | 1) The cry for individual operational tools as an argument for individual self-built solutions is not valid, because in-house development is usually not a viable alternative – also, professional software vendors such as minubo can easily develop custom tools for single customers as an individual supplement to their standard frontend (see, for example, the story of the Curated Shopping startup SugarShape). 2) The demand for individual visualization and flexible analysis, however, is valid, but covered by any good standard software in one way or another.
2. Data Model: Metrics, Entities, Attributes, Logic
Questions | Which key figures, entities and attributes are there? How are they defined or calculated - and how are they logically related? Does the terminology used fit my own?
The Facts | No question: the data model is the linchpin of what a BI solution can achieve. By defining the structure for measures, entities, attributes, and logic, it determines what can actually be analyzed and how. But whoever now thinks: "Ha, that’s what I am saying, that's exactly why I need an individual solution with an individual data model!" is still wrong, because within a field like retail, the structures of what constitutes the activities of individual companies are so similar that a standard solution still is the better alternative - if it specializes in the respective field and is not just a generic analysis solution. Let me illustrate this by the following example: The most important entities in retail are customers, products and orders - accompanied by supplementary entities such as campaigns, purchase orders and distribution channels. Performance evaluation of these entities provides the required metrics (around the core of transaction metrics as the centerpiece of any analysis of trading models) as well as the required attributes as feature categories of the structural entities. From all this, the (at least) 80% overlap of the requirements of retailing companies in business intelligence emerge. Only the remaining 20% result from a) special business models such as marketplace models or curated shopping, b) special processes such as different logistics models and/or c) individual metrics terminology. In fact, good, well-established BI software like minubo covers a large part of a) and b) despite its standardization approach, because through their broad customer portfolio respective extensions to the data model have been built up over time - and what is not already covered can still be implemented as an individual or general extension to the data model. c) is a subject of its own, which I will be treating separately here in the blog soon; In summary, I can say: Yes, wanting to implement your own metrics terminology is fair enough, but the sad truth usually looks different for such corporate terminology. In most cases, each department maintains their own terminological habits, so cross-departmental communication becomes a problem, and often, such systematics are even lacking completely. (We tackle this problem with the "Commerce Reporting Standard" project - just have a look.) So, in fact, a truly consistent terminology that comes with professional BI software really can only push companies forward; In the rare cases where companies actually have consistent terminology of their own, alias names for metrics and attributes can be set up in most BI software.
Conclusion | Yes, the actual individualization requirements are usually hidden right here - but field-specific BI software can generally cover them well, too, be it directly in the standard setup that has grown according to broad customer portfolios or as an individual customization or extension of the standard scope.
3. ETL: Data Transformation
Questions | How is my data processed on the way from the source system to the data warehouse? Do I have access to the processing logic and can I customize it?
The Facts | The fact is, the better (and more stable in terms of long-term usability) the data model of a solution is, the less ETL processes need to be adjusted after the initial setup. Sure, source systems can change and make adjustments to the ETL process necessary, but that's a maintenance issue, not requirement-based. The case on this level of individualization is therefore very simple: Yes, companies often feel the need to be able to access the data processing themselves, but with good solutions that have sophisticated data models, this is basically not necessary. Anything that may come up here as customization requirements from a business point of view should actually be more a question of the data model and make adjustments to the ETL necessary only as a consequence of that. The fact that the need for direct access in the minds of many people still exists, seems to me mainly due to the habit or the fact that BI self-construction projects often end up as a mere collection of ETL processes, so that an unjustifiably strong focus lies with this solution component, which is actually justified only from a technical point of view, not from the business perspective.
Conclusion | From a business point of view, customization requirements within good BI solutions are hardly justified on this level; from a technical perspective, they are just important for those who need to host and maintain the solution - and for purchased solutions this is not the client company itself.
4. Integration: Data Sources and Third-Party Systems
Questions | Which data sources can I connect to my business intelligence - can I get all the data points into the DWH that I need? Can I use my BI-consolidated data for my processes in third-party systems (e.g., email marketing)?
The Facts | In fact, the question of integrable data sources is also primarily a matter of the solution scope at the data model level. What is included here in terms of data record types can generally also be integrated on the source side - if not via a standard interface to the specific system, then via generic interfaces that require the one-time establishment of data exports in a specific format. Such an interface is part of every good BI software. Thus, this aspect also refers us back to the remarks in point 2. With regard to outgoing interfaces for the utilization of consolidated data in third-party systems, I would again like to state that we are dealing with a basic requirement of modern business intelligence solutions here, because their purpose is not primarily about producing colorful pie charts, as it was in earlier times, but to a large extent about making operational processes intelligent and, if possible, automating them. For this purpose, outgoing interfaces that access the entire database at the raw data level and can be flexibly configured are absolutely necessary. If you buy a modern BI software, you should not encounter any problems here and can easily set up flexible data transfers for individual processes wherever you want them. Therefore, individualization requirements don’t have any room in this area neither and, again, do not provide an argument for a self-built BI.
Conclusion | Similar to point 3, there are no valid individualization wishes at this level either. Again, they take us back to point 2 or become void as long as the sales dialogue takes place with providers of actually modern BI solutions.
From the statements made in the paragraphs above, it becomes clear that individualization wishes in business intelligence (unlike the corresponding communication with companies often suggests), on closer inspection, only hold true at two levels:
- For the most part on the data model level, which in turn has (purely technical!) effects on source system connection and ETL processes. These individual requirements, however, can generally be well met by field-specific solutions (e.g., by minubo for the commerce sector) - either with a standard solution that has grown accordingly based on a broad customer portfolio or through individual adjustments or extensions to the standard setup.
- To a lesser extent, at the level of individual frontend tools beyond visualization and analysis. However, since companies in self-built scenarios are almost never able to build such frontends themselves, this is anything but an argument for self-construction, but only the right incentive to buy the solution of a professional provider, who is able to extend their standard frontend with individual tools – for example, minubo works with plugin technology for this purpose.
The only knock-out case in the context of individualization wishes in BI (always assuming we speak of a truly powerful, up-to-date BI software) is the "out of scope" scenario: primarily at the data model level, but in some cases at the frontend level, too, a solution provider may have to argue that the needs of a company are simply not in the field that they want to cover with their solution. In this scenario, there are only three viable ways: 1) The provider is convinced to follow the suggested path by a very good financial offer; 2) the purchased solution is supplemented by homemade components; 3) the "out of scope" requirements are so extensive that the effort for a DIY solution is actually worthwhile.
Despite this potential case, the conclusion of this article remains: Since the individualization wishes of companies for their business intelligence are mostly felt (and communicated) in a disproportionately strong way, but in fact only play a role on the two levels mentioned above and can be covered by a good BI software on both levels in most cases, every company should think twice about whether the immense amount of money and time required for self-built BI is actually worthwhile based on the idea of individualization - because this conclusion is unfortunately still drawn too fast based on what we have seen to be a rather unstable foundational premise of individualization wishes. On this topic, I would also like to refer to my last blog post: “Make or Buy” in BI? The truth is, "buy" provides the better solution - faster and cheaper.
As always, I am looking forward to personal communication on the subject - just contact me at email@example.com and I'll get back to you as soon as possible.
If you would like to be informed about this and other articles, please sign up for the automatic email update - there will be a new post approximately once a month.
Everyone is talking about data. Data is the new currency, data decides success or...
Again and again, I cannot decide whether to scream or cry – or just leave the meeting...