DEVOCEAN

Abstract

The project involves developing a converged application server that acts as a bridge between HTTP-based microservices and the SIP (Session Initiation Protocol) used by an IMS (IP Multimedia Subsystem). The server receives HTTP requests, converts them into SIP messages, and manages SIP registration with the IMS. The IMS provides an Expiry header (e.g., 8400 seconds) indicating the interval for SIP registration refreshes.

A key challenge arises during server deployments or restarts due to the absence of active-passive server structures, which can interrupt or result in the loss of local timers responsible for managing SIP registration refreshes. This interruption can cause missed registrations, leading to lapses in SIP registration and potential service disruptions. A possible solution could have been manually triggering registration in batch for all the users for lost registration. But this may not be an effective solution when we are dealing with millions of users.   In this article, we will explore how refreshing timers' persistence is preserved through Redis key event notifications and how the refresh cycle of SIP registrations can be maintained reliably.

To mitigate this, a resilient mechanism must be implemented to ensure continuous SIP registration, even during server reboots or redeployments. Additionally, it is crucial to note that all SIP registration servers are stateless by design, and a round-robin strategy has been adopted to efficiently handle incoming HTTP requests as users join or leave the services.

SIP Registration in IMS

SIP registration in A.(에이닷) with IMS on behalf of UE

Address the challenges of losing SIP registrations

To maintain uninterrupted SIP registration and address server restart challenges, we use Redis key event notification for persistent timer callbacks

• Persistent Timer Management:

  • Upon receiving an HTTP API call, the server performs SIP registration with the IMS and retrieves the Expiry header (e.g., 8400 seconds). It sets a persistent timer in Redis using the SETEX command, with a primary timer key calculated as (Expiry/2) + delta(A), where delta Is a random value between 0 and 300 seconds whereas the delta is adjusted as runtime configuration through a configuration server.A secondary timer key is set to (Expiry/2) + (Expiry/4) + delta(A) seconds to account for potential delays during server bootup , providing a fallback to catch missed keyspace notifications and maintain SIP registration.

• Redis Keyspace Notifications with PSUBSCRIBE:

  • The server subscribes to Redis keyspace notifications using the PSUBSCRIBE , allowing it to listen for key expiration events

• Handling Server Reboots:

  • On reboot or redeployment, the server restores its subscription information and reinitializes the subscription channel in Redis. The secondary timer key provides a buffer period to capture any missed notifications during startup.

• Scalability and Redis Cluster:

  • Redis Cluster is used for scalability, supporting horizontal scaling by distributing data across multiple nodes. This ensures efficient handling of growing user loads and high request volumes, with automatic sharding and fault tolerance, making it more suitable than Redis Standalone or Redis Sentinel for large-scale deployments.

By integrating Redis keyspace notifications with PSUBSCRIBE and adopting Redis Cluster for scalability, the proposed solution effectively maintains SIP registration across server reboots and redeployments. This approach ensures reliable SIP registration management ensuring continuous service without manual intervention & addressing the core challenges of maintaining persistent SIP registration.

Redis Key Expiration Events

Before proceeding further, we can look at an important Redis feature that works regardless of the type of value we are storing: key expiration. Key expiration allows users to assign a specific timeout, commonly referred to as "time to live" (TTL), to a key. Once the TTL period concludes, the key is automatically removed from the Redis database.

A few important notes about key expiration:

• Expiration times can be specified with either second-level or millisecond-level precision,the expiration time resolution is always 1 millisecond

• Information regarding key expirations is replicated across Redis instances and persisted to disk. Notably, if the Redis server is halted, the passage of time does not advance virtually. Instead, Redis preserves the exact timestamp at which a key is scheduled to expire, ensuring accurate expiration behavior upon server restart

Use the EXPIRE command to set a key's expiration in redis-cli:

> set key 100 ex 10
OK
> ttl key
(integer) 9

Redis Subscription for Keyspace Expiry Notifications

Inspection through redis-cli monitoring tools, let's see the hash slots in the cluster for example

> cluster slots
1) 1) (integer) 0
   2) (integer) 5460
   3) 1) "172.23.xxx.xx1"
      2) (integer) 6379
      3) "235a0359fe82e21dffcfc2d8d2ef84ac325c1a8e"
   4) 1) "172.23.xxx.xx2"
      2) (integer) 6383
      3) "b79e85c0835bed56ee1163e169afc56e47b69adc"
2) 1) (integer) 5461
   2) (integer) 10922
   3) 1) "172.23.xxx.xx3"
      2) (integer) 6380
      3) "9b8e39817cb4fa7a7e81750d4875e4968cf1f8fb"
   4) 1) "172.23.244.226"
      2) (integer) 6384
      3) "5c094e567603dec420b32916e70fbf5508ef7171"
3) 1) (integer) 10923
   2) (integer) 16383
   3) 1) "172.23.xxx.xx4"
      2) (integer) 6381
      3) "d48b1b414db2b19c58add9b50c89a443375b6b3d"
   4) 1) "172.23.244.226"
      2) (integer) 6382
      3) "0a5dc3a38e87b2620b4323ad3bc5542ef0724f1a"
>

Subscribing to redis for receiving expiry events in the application

Events capturing using redis-cli monitor

redis:6381> PSUBSCRIBE __key*__:expired
Reading messages... (press Ctrl-C to quit)
1) "psubscribe"
2) "__key*__:expired"
3) (integer) 1
1) "pmessage"
2) "__key*__:expired"
3) "__keyevent@0__:expired"
4) "{***********}:REG_EXPIRY:172.27.0.222"
 
+------------------------+
| Enable Keyspace |
| Notifications |
| (CONFIG SET |
| notify-keyspace-events Ex) |
+-----------+------------+
    |
    |
    v
+------------------------+
| PSUBSCRIBE to |
| "__key*__:expired" |
+-----------+------------+
    |
    |
    v
+------------------------+
| Wait for Expiration |
| Events |
+-----------+------------+
   |
   | Expiration
   | Event
   v
+------------------------+
| Handle Event: |
| Example: |
| {"type": "pmessage", |
| "pattern": "__key*__:expired",|
| "channel": "__keyevent@0__:expired",|
| "data": "mykey"} |
+------------------------+

Persistency in key notifications with Redis

Organizing Key Pattern

In a distributed environment where multiple server instances subscribe to Redis keyspace notifications using the pattern keyevent@0:expired, all servers receive notifications for any key expiration. Redis does not support patterns with wildcards for keyspace events, necessitating a method to filter and handle the events correctly.

Key Pattern and Subscription Handling

• Subscription Pattern

  • Each server instance subscribes to the keyevent@0:expired pattern to receive notifications for key expirations. all subscribed servers get the notifications, necessitating a method to filter and handle the events correctly.

• Key Naming Convention

  • A key pattern that includes the server's IP address is used: {*************}:REG_EXPIRY:172.27.0.222. This key name uniquely identifies the server responsible for handling the expiration event.

• Event Filtering

  • When a key expiration event is received, each server extracts the IP address from the expired key name. If the extracted IP matches the server's IP address, the server processes the event.

• Fast Ignore Mechanism

  • Servers whose IP address does not match the one in the expired key name perform a "fast ignore," quickly dismissing the event from the application core layer to optimize performance and reduce unnecessary processing.

By implementing this key-naming strategy and filtering mechanism, we ensure that only the designated server processes the expiration event.

Why Use Secondary Keyspace Notifications

Keyspace notifications allow clients to subscribe to Pub/Sub channels to receive events affecting the Redis data set.

Examples of events that can be received are

• All the commands affecting a given key.

• All the keys receiving an LPUSH operation.

• All the keys expiring in the database 0 in Redis cluster mode.

Redis Pub/Sub is fire and forgets that is, if your Pub/Sub client disconnects, and reconnects later, all the events delivered during the time the client was disconnected are lost. Hence during a server is redeployed and restarted, until the server re-subscribes it , during the booting process of the server, all incoming key events are lost. hence the secondary key would be a good choice to set

Events in a Redis cluster

Each node within a Redis cluster generates events pertaining to it's specific portion of the keyspace, as outlined previously. Unlike standard Pub/Sub messaging across the cluster, event notifications are not broadcast to all nodes. In other words, keyspace events are confined to individual nodes. Consequently, for clients to capture all keyspace events within the cluster, they must subscribe to every node independently. Our solution adheres to this approach by subscribing to each node separately. This strategy also proves advantageous during failovers, as it ensures that the system remains functional even when a master node fails and the cluster's topology is updated.

A goalkeeper in the system for SIP registration resilience

In a distributed system managing SIP registrations, component failures can lead to the loss of critical registrations. For instance, if a registration server instance for a specific SIP registration fails to refresh it, the registration might lapse since no other server processes the expired keys corresponding to different IP addresses. To mitigate this risk, a "Goalkeeper" mechanism is introduced to ensure resilience and recovery in such scenarios where new recovery requests would reach the available instances.

One of the solutions we initially considered was the use of a secondary key to create a more generic mechanism for handling expired registrations. The idea was that if the secondary key expired, another server could take over and process the refresh registration. This approach would distribute the responsibility of maintaining registrations across multiple servers, potentially reducing the burden on any single server.

However, we decided against this approach to avoid the complexities and potential pitfalls associated with holding distributed locks & especially the race algorithms.

Currently, the primary burden on the Goalkeeper is ensuring that it is notified of each refresh registration, including the refresh time and the expected expiry time. This notification process is crucial for maintaining accurate tracking and ensuring that no registrations are missed if one of the registration servers is taken down or any hw failure

The Goalkeeper server acts as a safety net, monitoring for missed SIP registration refreshes. It ensures that even if a primary server fails, the necessary registrations are maintained by checking the timestamps of the last successful refreshes and comparing them against the current time and expected intervals.

1. Registration Monitoring

   • The IMS provides an expiry header value Xs in the SIP registration response, which defines the maximum duration for which the registration is valid (𝑍𝑎𝑏𝑠 i.e 𝑇𝑙𝑎𝑠𝑡 + 8400s). So now, if the expiry is 8400 seconds, the Registration server is expected to refresh the registration before to IMS by

     • 𝑅𝑇𝑚𝑎𝑥(𝑀𝑎𝑥𝑅𝑒𝑓𝑟𝑒𝑠ℎ𝑇𝑖𝑚𝑒) = 𝑥/2 + 𝑥/4 + 𝛿(A)

       • For example 𝑅𝑇𝑚𝑎𝑥 = 8400/2 + 8400/4 + 𝛿(200) = 6500 𝑠𝑒𝑐𝑜𝑛𝑑𝑠 ,This means the registration should be refreshed no later than 6500 𝑠𝑒𝑐𝑜𝑛𝑑𝑠 after the last successful refresh by the SIP Registration server, assuming for example the primary delta is 200s.

2. Expected Expiry Time

   • As the IMS side expects a fresh registration request before the registration expires (i.e before 𝑍𝑎𝑏𝑠). So If the last successful refresh occurred at 09:00 AM, the maximum time the registration is valid is: 09:00 AM + 8400 seconds, This gives a deadline of 11:33 AM~, which is the absolute latest time by which the registration must be refreshed to IMS. The Goalkeeper must ensure that the refresh happens before this time in case of any failure, usually the rarest.

3. Monitoring Schedule

   • As the Registration Server is expected to finish the refresh with 𝑅𝑇𝑚𝑎𝑥𝑡𝑠 i.e

     • 𝑅𝑇𝑚𝑎𝑥𝑡𝑠 = 09:00𝐴𝑀 + 6500 𝑠𝑒𝑐𝑜𝑛𝑑𝑠 = 10:48𝐴𝑀`

4. Checking for Missed Registrations

   • The Goalkeeper checks for missed registrations by verifying if the difference between it exceeds the expected refresh time by 𝑇𝑐𝑢𝑟𝑟𝑒𝑛𝑡 − 10 𝑚𝑖𝑛𝑢𝑡𝑒𝑠 > 𝑅𝑇𝑚𝑎𝑥𝑡𝑠

With an example

• Xs: Expiry time (in seconds) provided by the IMS in the SIP registration response.

• 𝑇𝑙𝑎𝑠𝑡 ​: Timestamp of the last successful refresh.

• 𝑍𝑎𝑏𝑠=𝑇𝑙𝑎𝑠𝑡 + 𝑋𝑠 So this absolute expiry Time, where refresh should be triggered to IMS.

• 𝑇𝑐𝑢𝑟𝑟𝑒𝑛𝑡: Current time.

• 𝑇𝑏𝑢𝑓𝑓𝑒𝑟: Buffer time, set to 10 minutes (600 seconds) for example, Configurable at runtime.

• 𝑅𝑇𝑚𝑎𝑥 = Xs/2+Xs/4+𝛿(A) : Max Refresh Time by sip registration server .

• 𝑅𝑇𝑚𝑎𝑥𝑡𝑠 = 𝑇𝑐𝑢𝑟𝑟𝑒𝑛𝑡 + 𝑅𝑇𝑚𝑎𝑥

• 𝑇𝑐𝑢𝑟𝑟𝑒𝑛𝑡−𝑇𝑏𝑢𝑓𝑓𝑒𝑟 > 𝑅𝑇𝑚𝑎𝑥𝑡𝑠 : Checking Condition to determine if the refresh has been missed:

• Scenario

  • Xs: 8400 seconds

  • 𝑇𝑙𝑎𝑠𝑡: 09:00 AM

  • 𝑇𝑐𝑢𝑟𝑟𝑒𝑛𝑡: 11:05 AM

  • 𝑍𝑎𝑏𝑠 = 11:33~ AM

  • 𝑇𝑏𝑢𝑓𝑓𝑒𝑟 : 10 m ( 600 sec) in this example.

  • 𝑅𝑇𝑚𝑎𝑥ts=09:00 𝐴𝑀 + ( 8400/2 + 8400/4 + 𝛿(200) = 10:48 AM

  • 𝑇𝑐𝑢𝑟𝑟𝑒𝑛𝑡−𝑇𝑏𝑢𝑓𝑓𝑒𝑟 > 𝑅𝑇𝑚𝑎𝑥ts = (10.55 AM >= 10:48 AM).

The condition is valid. This indicates that the registration has not been refreshed as expected, and the Goalkeeper accounts for it and takes necessary actions, generally expected to happen if any hw failure as such. This mechanism ensures that even if it fails to refresh the registration on time, the Goalkeeper steps in to maintain the registration, preventing service disruption. The 10-minute buffer serves as a safety margin to ensure that any delays or issues are accounted for, allowing the Goalkeeper to act before the registration fully expires at the maximum limit set by the IMS.

How it works

The SIP registration service receives registration requests from a goalkeeper a.k.a subscription. Each time the registration server successfully registers with the IMS (IP Multimedia Subsystem), it notifies the goalkeeper with the relevant response code and the next scheduled refresh interval. The goalkeeper keeps a comprehensive record of these registrations for accounting purposes and conducts background scans to detect any missed refresh registrations, taking corrective actions as needed. Such missed registrations are rare, typically occurring due to hardware failures on the server side. In general deployment scenarios, it is unlikely that secondary key event notifications are missed, as the secondary key expiration is configured to account for the maximum boot time of the server.

Redis keyspace expiry delay with large user numbers

To effectively scale the system for managing a large number of user registrations considering the keyspace expiry notification event delays, For example, performance testing conducted on a system with a 4-core CPU, 8GB of RAM, and a 3-node Redis master-slave cluster demonstrated that, when handling over 1 million active registration keys, key expiry notifications were delayed by an average of 200 seconds. However, given the nature of our application, which includes a registration refresh window of 4200 seconds with an additional random delta to flatten the incoming request curve for the concurrency of new user join requests. Such delays may not significantly impact considering the nature of our application which does not need to be on real-time refresh where a delay of seconds to minutes is reasonably affordable. Even with an expected delay of 6 minutes for up to 2 million users, the system would still maintain adequate registration refresh cycles. On average, we can expect keyspace expiry notification delays for 1.5 million users as follows hence it would be a choice that we consider further scaling Redis clusters by adding more nodes.

However, when scaling to a large user base, such as 20 to 50 million users, the delays associated with Redis keyspace events must be carefully considered to ensure system reliability. When using a Redis cluster, scaling horizontally by adding additional Redis nodes within the same Redis cluster is like a solution. Still, it is important to note how Redis keyspace expiry notifications are broadcast to all consumers subscribed to the keyspace, regardless of specific key patterns.

• To enhance delay , we propose also to adjust hz config of redis nodes..

  • hz 10: Default. Good balance between responsiveness and CPU usage.

  • hz 50: Increases responsiveness at the cost of very little higher CPU usage.

  • hz 100: Even more frequent background tasks, may be useful for very high-throughput environments.

One can set hertz config through redis-cli as well by the below command or through redis config file.

CONFIG set hz 50

We have observed a significant improvement in key expiry notification delays with hz 50 in the following statistics when having a performance test for 500,000 thousand users taking 10 minutes of sample data of refresh registration cycle. One can also go ahead with higher values like 100.

Conclusion

Redis keyspace event notifications have markedly enhanced the reliability and resilience of our SIP registration system by effectively addressing challenges associated with server restarts and interruptions. This implementation enables persistent timer management, ensuring that SIP registration refreshes remain consistent even during server reboots. The inherent scalability of the Redis Cluster allows for the efficient handling of large volumes of user registrations while maintaining high availability, a critical requirement for robust service delivery.

Looking forward, we are exploring the development of a persistent scheduler to minimize dependencies on Redis keyspace notifications. This initiative is currently in the research and development phase, focusing on alternative approaches for scheduler management in distributed systems subject to handling millions of key TTL events, to identify optimal strategies for enhancing our system's functionality further. These advancements could lead to even greater resilience and flexibility, enabling us to proactively respond to the evolving demands of our infrastructure.

We encourage all to stay tuned for further developments in this area, as our R&D efforts continue to prioritize the enhancement of our services including the SIP registration system's overall effectiveness, for millions of users.

We thank you for your valuable time in reading this article. Any comment is welcome and would love to take it on a positive note for us as we learn together and grow.