The building blocks of a pricing engine
Written by Eddie O'Donnell
At the core of Zego's products is the pricing engine. It determines the risk associated with each customer requiring insurance and delivers the price attached with this risk. How does it do that?
In this blog post, we will explore how and why the pricing engine is designed as it is. We will also have a look at some challenges the Pricing & Risk engineering team face, and talk a bit about the future of pricing at Zego and the insurance industry as a whole.
In the insurance industry, one of the most common ways to model risk and to therefore calculate the total price of an insurance product is to use a generalised linear model (GLM). At Zego, we combine GLMs with other, more sophisticated models to create our pricing engine. However, for the purpose of this blog post, let's assume that we use GLMs alone.
An insurance product's price for a particular person is directly correlated to how likely a user is to make a claim, and if a person does make a claim, how severe it would be. This is a person's risk. The art of insurance is to be able to calculate this risk as accurately as possible, and a GLM is an industry standard. Essentially, a GLM is a function that takes in data to do with a person, such as their postcode, age or car, and outputs an associated risk. When calibrating this model, risk factors are calculated, which gives a good way to see how parts of a person's data contributes to their overall risk. It is these risk factors which are modelled in the Pricing Engine at Zego, and this blog post will attempt to explain how this has been engineered in our Python backend.
Engineering Risk Factors
To be able to engineer a GLM, two core versatile classes are required:
- Risk Factors
- Risk Rules
These two classes correlate to the risk factors and risk likelihood/severity respectively described in the previous section, so each risk factor will have a host of associated rules (which are key-value pairs of value → risk). The pricing engine will be able to generate a price for a particular person by multiplying the list of resulting risk rules to the base price of a product.
As an example, suppose Zego's actuaries have decided that for a particular product, based on their research and calculations, that the material of a vehicle's body was an appropriate risk factor for a particular product (fire engine insurance). In this case, certain materials would increase the driver's risk of having a severe claim, whereas others materials would lessen the chance of having a severe claim.
The engineering implementation would involve creating a new risk factor in our Python Django backend which represented the VehicleBodyMaterialFactor .
The inherited RiskFactor has lots of fields and methods which each individual risk factor can override. For example, the data_source field describes what type of value the field will match on. A DiscreteRule represents an exact match is required for the rules in the database, but for other risk factors, numerical values that take on a wide range of values may be used. It would be cumbersome to have a rule for each possible numerical value in a range, and so a NumericRule is used in the database which only represents the risk factors of the boundaries of each group.
We can then test this risk factor in order to mimic the requirements of the actuaries exactly, enabling us to be sure that their logic has been implemented correctly.
The actuaries could then insert the correct rules into our database (e.g Carbon fibre → 1.5x, Wood → 2x, Cast Iron → 1.2x) which mimic the risk of having a severe claim with those vehicle materials, but actually acting as a multiplier to the price of the product. The actuaries can then decide which products to place this new risk factor into, all using the Django admin site.
At Zego, we are striving to provide the fairest pricing possible. This means pricing each person uniquely, relating to data which reflects a person's individual risk in the most appropriate way. Our actuaries and data scientists are leveraging telematics and work provider data, to name a couple, in order to find ways of pricing people in novel ways which break from the norm of static risk association. See this blog post for more information.
On this note, we are now becoming more interested in creating novel risk factors which require considerably more mathematical rigour, driven by insights gathered from informative data sets generated directly from Zego's individual customers. Crucially, some of these novel risk factors require more specialised data processing and modelling in order to output accurate likelihoods. The base RiskFactor class shown in the example above provides plenty of methods to override which allows for a good amount of data enrichment in our Django monorepo, however some risk factors require their own space for processing and testing the large amount of data, as this could increase the latency for a quote considerably if the processing was done within the RiskFactor class. It also makes sense for some of these risk factors to have a specialised toolset to be able to, for example, verify the results of the productionised models are as expected.
Therefore, for some of our novel rating factors, a lightweight microservice has been created in order to receive data specific to the risk factor (for example, work provider shift data over the past month), process the data and return a multiplier to be used in the price of a quote.
This example shows how the RiskFactor class has been modified in order to call the endpoints of the microservices but still look like a normal risk factor (i.e. returns all the same objects). This means that each quote will still get details about each individual novel risk factor, just like the rules do in our database.
We can see in this code snippet that as the microservice is supplying the multiplier instead of well defined rules mapped out by the actuaries in the database, something which looks like a rule, but isn't, is being returned by the fetch_rule_from_fields method. This allows for this system to integrate smoothly with everything else that this risk factor is in contact with.
Adjusting for Performance
One issue with making complex risk factors into their own microservice is that latency starts become noticeable for each quote. We want to keep this number as low as possible, because no one likes waiting for a page to respond. We use Datadog for a variety of monitoring and logging purposes and one feature which has been very useful has been tracking the latency of individual function call in the code base. It was obvious that sending requests to an external service to get a multiplier for a rating factor was going to take considerably more time than just hitting our database for a multiplier. This flame graph shows exactly how much more time costly this approach is.
You can see that beneath the get_premium function, the green predict method takes up about the third of the time of that call, whereas all the yellow bars (which represent all the other risk factors and are database-only calls) take a fraction of the time.
It is apparent that in order to scale this solution, the calls out to the novel rating factors have to be asynchronous. Having just upgraded to Python 3.8, we have been blessed with many performance benefits as well as syntactic sugar when working with Python's asyncio package in particular. Database calls in threads in Django are unsafe and so in order to make this work, the external calls need to run before or after the database queries. In an ideal world, this would be the structure of the pricing engine request:
However, we must settle for this setup which will only give us performance benefits for more than one novel rating factor:
We are in the process of making the pricing engine its own service, separating it from our current monolith. This will allow it to grow at its own rate and become a much more readily available service to everyone who needs to use it. For example, this would allow the pricing team to test updates to the pricing models in a safe environment, by forwarding requests from the main service onto the test environment.
To enable data science to develop novel rating factors faster and test them more thoroughly, it may also be worth combining their individual services into this future pricing engine service. This will allow them to unit test their features in the same manner as the software engineers do, but also enable good integration testing and also reduce latency considerably.
Finally, there is potential for a complete rethink of the underlying risk-factor-rules system that the pricing engine is currently built on. For example, we are in the process of creating a purely machine learning based pricing model to see how this would perform in replacement of our current system. This would allow potential for more accurate pricing, but may hinder transparency to the customer.
Zego strives for accurate, fair and transparent pricing, and this will always be at the forefront of what we do.