Unified real-time billing system for major telecom

Telco Billing system is the core around which all other services are built. These systems come in all shapes and sizes and more often than not are developed by external, highly specialized vendors.

Unusually, the operator we worked with was using an internally developped post-paid solution. The system worked, but the market at that time was moving towards prepaid and multi-service offerings, including retail and corporate plans with split billing and complex tariff structures. Also, the compliance requirements were evolving fast.

The problem

Our team was called upon to start from an existing solution and to smoothly introduce all new features, ensuring continuity and rollback capability at all times.

Core requirements:

• Real-time service access control, with precise zero balance call cutoff.
• Telco grade availability with graceful fallback to legacy billing system even in case of catastrophic multi-point failure.
• Runtime adjustable / deployable complex call rating and charging rules.
• Fully reversible charging logic, supporting zebra functions, time based split billing etc. Need to compute both the cost of X minutes, as well as how much time Y$ buys.
• Support for convergent billing of mobile, VoIP, fixed telephony, optical internet, Satellite TV, IPTV etc.
• Explainable bill generation with fully customizable tariffs.

The outcome

We designed, built and deployed a unique convergent billing system that is still in operation today over 15 years later, having directly earned billions of dollars.

The tool we built gave the operator massive advantage which the competitors never closed. In addition, for several years, our system was used by HP to advertise their OCSAC platform to other customers by bringing them on-site.

The solution

The most critical feature of the new system was reliability and predictable scaling: using the same servers, our system went from a handful of QPS of the postpaid traffic to multiple thousands of QPS of the real-time prepaid. The 99.99th percentile latency was under 100ms and our novel custom solution on top of Tibco Rendezvous enterprise message bus provided carrier grade availability.

A special requirement for the switchover logic was the need to support an additional degraded mode, where HP OCSAC would allow all connections, while the offline (post-paid) billing system would take access control back over. This would be done by directly programming the switch (HLR) to disconnect users with negative balance.

Availability was further improved by hard separation of real-time / non-realtime parts and explicit scheduling-based resource management inside the java binary of the Prepaid System server (PPS core on the image).

The customizability of the system was facilitated through integration of DRools rules engine. In addition, run-time bytecode compilation of rating functions was supported for the cases when even DRools was not flexible enough. The package containing pricing tables, rating functions, rules and the rest of tariff plan configuration was built using custom GUI tools and tested on non-production instances by a dedicated team. The upgrade package was then rolled out to the production prepaid system core servers and activated by a built-in scheduler.

Because of the tight integration of TimesTen with Oracle RDBMS, the system was able to serve up to date balance information from the user interface for the reps and call center, IVR, USSD and all other downstream systems without taxing the real-time part.