Dear Customer,
We would like you to know about the following unplanned outage.
Service Affected: As our Primary geographic location is having issues, all traffic is being rerouted to our Secondary Geographic location.
Impact: Severe Impact
Start Date: Monday, 5th of February 2024, 10:12 AM (BST)
End Date: Monday, 5th of February 2024, 11:05 AM (BST)
Status Report: Service Status
Status: Ongoing, under investigation
Description:
We’re seeing possible issues with calls regarding our Primary Geographic location.
Our Geographic redundancy should mean that call processing is automatically transferrred to our Secondary, backup location but as this hasn't happend we have manually done so.
Our tech team, together with the manufacturer of the base solution, are investigating both the issue with the Primary node and why the auto fall-over did not kick in.
a) If you have set up your devices/systems according to our recommened setup, yor phones and systems should still be online and fully functioning.
b) However, if you only set up with the primary SIP Server Inbound calls may not process properly for some numbers and therefore not reaching you. This may also be the time to adjust your device/system settings to the recommended settings.
We're investigating what is happening and how to solve it asap.
All other services are unaffected.
Our apologies for the inconvenience this may cause and if there are any questions, please contact our technical support team.
Kindest Regards
the Voyced Technical team
Update 11:05:
Our tech team has found the issue and has corrected it, so all should start to be well again.
We're testing further to make sure all is indeed well again and will give further updates here.
Update 12:15:
The patch seems to have worked and we are now waiting for the 'old'/ongoing traffic on the Primary node to finish/time out so we can restart it and check if all calls are being processed correctly again.
Update 12:45:
All calls seem to go through properly again and both geographic areas are back online.
We'll keep monitoring to make sure all is well and if you still notice issues please let us know so we can look at those individual cases.
Again, apologies for any inconvenience and we'll work together with the manufacturer of the base solution to understand why and how this happened today and how we/they can adjust things to prevent this from happening again in the future. Especially why the Automatic Geographic Redundancy did not automatically kick all traffic over to the fall-back, the way it is meant to do.
Update 14:30:
We have heard that a few devices had an issue making and/or receiving calls.
This is (usually) solved by rebooting the device so that you force the device to re-register and therefore connect to the proper routings again.
Let us know if you are still running into issues after doing so.
Monday, February 5, 2024
