tag:status.twentythree.com,2005:/historyTwentyThree Status - Incident History2024-03-19T09:48:59+01:00TwentyThreetag:status.twentythree.com,2005:Incident/201055202024-03-13T22:44:48+01:002024-03-13T22:44:48+01:00Scheduled maintenance<p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>22:44</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>22:29</var> CET</small><br><strong>Verifying</strong> - The maintenance has completed. Some updates took longer than expected, with some functionality being unavailable for up to 30 minutes during the window. We apologise for any inconvenience caused.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>21:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>11:07</var> CET</small><br><strong>Scheduled</strong> - We will be updating different components of our infrastructure. We do not expect any major downtime across the platform, but different functionality of the platform will be limited for up to 10 minutes at a time during the maintenance window.</p>tag:status.twentythree.com,2005:Incident/201055112024-03-11T23:17:17+01:002024-03-11T23:17:17+01:00Scheduled Maintenance<p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>23:17</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>21:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>11:06</var> CET</small><br><strong>Scheduled</strong> - We will be updating different components of our infrastructure. We do not expect any major downtime across the platform, but different functionality of the platform will be limited for up to 10 minutes at a time during the maintenance window.</p>tag:status.twentythree.com,2005:Incident/201076582024-03-05T23:16:51+01:002024-03-05T23:16:51+01:00Scheduled maintenance<p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>23:16</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>22:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>15:50</var> CET</small><br><strong>Scheduled</strong> - We will be updating our “live01” server. During the maintenance window the server will not be available for up to 15 minutes at a time. As a part of the update, the IP of the server will change.<br /><br />Customers with live streams connected to "live01" during this window may have to restart the stream to ensure the stream reconnects correctly, if the encoder does not reconnect automatically.</p>tag:status.twentythree.com,2005:Incident/200959772024-02-27T21:30:00+01:002024-02-28T11:04:47+01:00Slow responses from the TwentyThree platform<p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>21:30</var> CET</small><br><strong>Resolved</strong> - On the 27th of February between 20:36 UTC and 20:39 UTC the TwentyThree platform responded with high latency due to a surge of traffic on the platform, which increased faster than the automated solutions for handling more traffic.<br /> <br />The higher latency affected the real-time parts of the system such as:<br /> * Admin interface<br /> * Engagement functionality on webinars<br /><br />It did not affect:<br /> * Video playback<br /> * Ongoing live streams</p>tag:status.twentythree.com,2005:Incident/199468012024-02-09T14:45:53+01:002024-02-09T14:45:53+01:00Webinar registrations from HubSpot workflows not working<p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>14:45</var> CET</small><br><strong>Resolved</strong> - The issue relating to processing webinar registrations from HubSpot workflows has been resolved.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>11:28</var> CET</small><br><strong>Investigating</strong> - Webinar registrations from HubSpot workflows are currently not working.</p>tag:status.twentythree.com,2005:Incident/199465642024-02-09T11:30:32+01:002024-02-12T16:03:06+01:00Slow responses from the TwentyThree platform<p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>11:30</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>11:30</var> CET</small><br><strong>Update</strong> - The issue has been resolved. We will provide more information about the incident in the post-mortem section.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>11:08</var> CET</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>11:02</var> CET</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>10:52</var> CET</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>10:45</var> CET</small><br><strong>Investigating</strong> - We are investigating an ongoing issue causing the TwentyThree platform to respond with more delay than usual.<br /><br />We will update this incident once we know more.</p>tag:status.twentythree.com,2005:Incident/199230322024-02-06T21:21:32+01:002024-02-06T21:21:32+01:00Google Firebase Realtime Database is reporting issues<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>21:21</var> CET</small><br><strong>Resolved</strong> - The issue has been resolved by the vendor.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>17:26</var> CET</small><br><strong>Investigating</strong> - Google Firebase Realtime Database is used in our webinar product to connect status between hosts and speakers. The service is reporting issues on their European cluster, https://status.firebase.google.com/incidents/aJrR8Pu9PFeepbgkWkLm, and we are monitoring status actively.</p>tag:status.twentythree.com,2005:Incident/198041432024-02-05T23:20:12+01:002024-02-05T23:20:12+01:00Scheduled Maintenance<p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>23:20</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>22:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>10:48</var> CET</small><br><strong>Scheduled</strong> - We will be updating different components on our infrastructure. Different parts of the platform will be severely limited for up to 10 minutes at a time during the maintenance window.</p>tag:status.twentythree.com,2005:Incident/198189722024-01-24T10:00:00+01:002024-01-24T16:31:46+01:00Slow responses from the platform<p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>10:00</var> CET</small><br><strong>Resolved</strong> - On 24th of January between 9:00 and 9:05 UTC the TwentyThree platform responded with latency going up to 25 seconds in some parts of the platform. The higher latency was caused by a large amount of traffic we were receiving at the moment.<br /><br />We noticed higher load at 8:57 UTC and provisioned more resources to the TwentyThree platform to alleviate the issue.<br /><br />We will be looking into provisioning more resources to appropriate components of our infrastructure to prevent similar issues in the future.</p>tag:status.twentythree.com,2005:Incident/197967112024-01-19T16:00:00+01:002024-01-22T18:51:37+01:00Platform not responding<p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>16:00</var> CET</small><br><strong>Resolved</strong> - On 19th of January between 14:56 and 14:58 UTC the TwentyThree platform was not responding. Our users would experience long loading time or getting error screens.<br /> <br />The outage was caused by an update on infrastructure, which disrupted connectivity between components, leading to a disruption of service. The update was previously tested and we did not expect any issues. We will investigate why the update did not go as expected, and adopt changes to prevent similar issues in the future.</p>tag:status.twentythree.com,2005:Incident/193677202023-12-06T11:00:00+01:002023-12-07T16:15:10+01:00Slow responses from the platform<p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>11:00</var> CET</small><br><strong>Resolved</strong> - On 6th of December between 10:00-10:10 UTC the TwentyThree platform provided degraded service. During the incident, a response to a request would be provided with a delay, and ~30% of requests resulted in an error response.<br /><br />The issue was caused by an unforeseen performance bottleneck. In response to the incident we are optimising parts of the system responsible for the bottleneck.</p>tag:status.twentythree.com,2005:Incident/190559742023-11-20T21:52:21+01:002023-11-20T21:52:21+01:00Scheduled Maintenance<p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>21:52</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>21:52</var> CET</small><br><strong>Update</strong> - Scheduled maintenance is still in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>20:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>12:45</var> CET</small><br><strong>Scheduled</strong> - We will be updating different components on our infrastructure. We do not expect any major downtime across the platform, but different functionality of the platform will be limited for up to 10 minutes at a time, while updating different components of the infrastructure.</p>tag:status.twentythree.com,2005:Incident/186890172023-10-09T22:03:18+02:002023-10-09T22:03:18+02:00Scheduled Maintenance<p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>22:03</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>21:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>16:28</var> CEST</small><br><strong>Scheduled</strong> - We will be updating different components on our infrastructure. We do not expect any major downtime across the platform, but different functionality of the platform will be limited for up to 10 minutes at a time, while updating different components of the infrastructure.</p>tag:status.twentythree.com,2005:Incident/185988212023-09-25T10:53:35+02:002023-09-27T11:31:53+02:00Slow responses from the the platform<p><small>Sep <var data-var='date'>25</var>, <var data-var='time'>10:53</var> CEST</small><br><strong>Resolved</strong> - Today between 7:55 and 8:05 UTC the TwentyThree platform experienced long response times. We are looking into the causes of the issue and we will update the incident once we gather more information.</p>tag:status.twentythree.com,2005:Incident/184797152023-09-12T23:00:20+02:002023-09-12T23:00:20+02:00Scheduled Maintenance<p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>23:00</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>22:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>14:45</var> CEST</small><br><strong>Scheduled</strong> - We will be scaling up parts our infrastructure to ensure the delivery of our normal quality of service.<br /><br />During the maintenance, parts the platform will not be available for 2-10 minutes at a time.</p>tag:status.twentythree.com,2005:Incident/183474982023-09-01T12:45:00+02:002023-09-01T12:52:58+02:00Platform connection issues<p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>12:45</var> CEST</small><br><strong>Resolved</strong> - Today between 7:37 UTC and 7:40 UTC we experienced a connectivity problem on the TwentyThree platform.<br /><br />For the duration of the issue, it was not possible to start webinars and some of the TwentyThree-hosted content was not available (depending on cache status on our CDN). The issue did not affect running webinars.<br /><br />Between 7:40 UTC and 7:49 UTC, we saw longer response times on the platform as it was recovering. This did not limit the platform’s functionality, but the platform would not provide our usual quality of service.<br /><br />The issue was caused by an update of the underlying infrastructure, which resulted in an unexpected disconnect between different components of the platform. In response to this issue we will be updating internal tooling and processes to prevent similar issues from happening in the future.<br /><br />We sincerely apologize for any inconvenience caused to our customers.</p>tag:status.twentythree.com,2005:Incident/175007342023-07-11T01:46:11+02:002023-07-11T01:46:11+02:00Scheduled Maintenance<p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>01:46</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>10</var>, <var data-var='time'>22:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>14:22</var> CEST</small><br><strong>Scheduled</strong> - We will be updating different components on our infrastructure. During the maintenance window, there will be 10-15 minute long intervals during which the platform will be partially unavailable.</p>tag:status.twentythree.com,2005:Incident/174764152023-06-12T21:13:04+02:002023-06-12T21:13:04+02:00Scheduled Maintenance<p><small>Jun <var data-var='date'>12</var>, <var data-var='time'>21:13</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>12</var>, <var data-var='time'>20:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>10:29</var> CEST</small><br><strong>Scheduled</strong> - We will be updating different components on our infrastructure. We do not expect any major downtime across the platform, but different functionality of the platform will be limited for up to 10 minutes at a time, while updating different components of the infrastructure.</p>tag:status.twentythree.com,2005:Incident/171092622023-05-13T13:43:43+02:002023-05-13T13:43:43+02:00Scheduled Maintenance<p><small>May <var data-var='date'>13</var>, <var data-var='time'>13:43</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>10:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'> 1</var>, <var data-var='time'>15:01</var> CEST</small><br><strong>Scheduled</strong> - We will be updating different components on our infrastructure. We do not expect any major downtime across the platform, but different functionality of the platform will be limited for up to 20 minutes at a time, while updating different components of the infrastructure.</p>tag:status.twentythree.com,2005:Incident/165261492023-04-09T00:15:27+02:002023-04-09T00:15:27+02:00Scheduled Maintenance<p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>00:15</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>19:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>11:33</var> CET</small><br><strong>Scheduled</strong> - We will be updating different components on our infrastructure. During the maintenance window, there will be 10-30 minute long intervals during which the platform will be partially unavailable.</p>tag:status.twentythree.com,2005:Incident/164006732023-03-18T10:55:48+01:002023-03-18T10:55:48+01:00Scheduled Maintenance<p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>10:55</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>10:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 9</var>, <var data-var='time'>13:26</var> CET</small><br><strong>Scheduled</strong> - We will be updating different components on our infrastructure. We do not expect any major downtime across the platform, but different functionality of the platform will be limited for up to 10 minutes at a time, while updating different components of the infrastructure.</p>tag:status.twentythree.com,2005:Incident/161762352023-03-11T12:37:01+01:002023-03-11T12:37:01+01:00Scheduled Maintenance<p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>12:37</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>11:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>13:46</var> CET</small><br><strong>Scheduled</strong> - We will be updating different components on our infrastructure. We do not expect any major downtime accross the platform, but different functionality of the platform will be limited for up to 10 minutes at a time, while updating different components of the infrastructure.</p>tag:status.twentythree.com,2005:Incident/163741242023-03-07T13:10:33+01:002023-03-07T14:57:25+01:00Slow responses from the the platform<p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>13:10</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>13:07</var> CET</small><br><strong>Investigating</strong> - We are investigating an issue on the platform causing slow loading. We will update this thread when we know more.</p>tag:status.twentythree.com,2005:Incident/161762212023-02-25T13:49:38+01:002023-02-25T13:49:38+01:00Scheduled Maintenance<p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>13:49</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>11:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>13:45</var> CET</small><br><strong>Scheduled</strong> - We will be updating different components on our infrastructure. We do not expect any major downtime accross the platform, but different functionality of the platform will be limited for up to 10 minutes at a time, while updating different components of the infrastructure.</p>tag:status.twentythree.com,2005:Incident/159200452023-02-04T13:00:27+01:002023-02-04T13:00:27+01:00Scheduled Maintenance<p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>13:00</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>11:01</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>11:53</var> CET</small><br><strong>Scheduled</strong> - On the 4th of February from 10:00 UTC, we will be updating different components of our infrastructure. We do not expect any major downtime accross the platform, but functionality of webinars will be slightly limited for a few minutes at a time during the maintenance window.</p>