AperNet

Happy New Year!

🎄

[Service Recover: Hong Kong Premium VPS] All influenced VPS had been migrated and powered up. We apologize for any inconvenience you may have experienced during this operation. We appreciate your long-term trust and support.

[Service Impacting: Hong Kong Premium VPS Migration] The NOC has identified hardware failure on a hypervisor hosting Premium VPS and dispatched repair. These currently-running VPS hosted on that hypervisor cannot be hot migrated to another hypervisor due to processor mismatch. We will shut down the VPS and then migrate the VPS to another hypervisor. This can cause downtime of up to 30 minutes. Your VPS will be powered on afterward. We apologize for any inconvenience you may have experienced during this operation. We appreciate your long-term trust and support.

[Service Recovery: Hong Kong] The NOC has identified the malfunctioning Ethernet switch and fixed it. Services are now back online. Some Premium and Elite Virtual Servers have been migrated and rebooted. We apologize for any inconvenience you may have experienced during this incident. We appreciate your long-term trust and support.

[Outage: Hong Kong] The NOC has observed an Ethernet forwarding issue and is currently diagnosing the root cause. The network performance might be influenced, and some services might be interrupted for a short time. We apologize for any inconvenience you may experience during this incident. We thank you for your long-term trust and support.

[Outage: Hong Kong Hypervisor Emergency Reboot] The NOC observed a fault daemon on one of the hypervisors hosting Premium Virtual Servers at 0825 and issued a reboot to the hypervisor. The hypervisor is now back online, and these affected virtual servers have been rebooted. We apologize for any inconvenience you may have experienced during this incident. We appreciate your long-term trust and support.

[Service Change: San Jose Residential VPS switched to Verizon] We have switched from AT&T to Verizon for our Residential VPS. Currently, the IPs assigned from Verizon has better overall quality. We have also disabled random source NAT by default. This feature is now available on-demand.

[Postmortem : San Jose VPS Server Outage Investigation] We have determined that the root cause of this outage is overheating during Debian update. We do not use closed-loop thermal control due to many reasons. Our monitoring service has been down since the last outage incident. The overheat caused a server shut down, and the subsequent boot partition corruption prevented the server from booting up again. Following corrective measurements have been applied: We have fixed the corrupted partition and initramfs, and set maximized cooling strategy for our server. We have also set up our monitoring service, with corresponding alarm settings.

[Outage Annoucement: San Jose VPS server down] With a heavy heart we announce that one EPYC server hosting San Jose VPS instances (purchased from new.apernet.io) is down. This is a hardware failure. We are still investigating the root cause and trying to boot up the server.

[IMPORTANT: End-of-Life Announcement for Tokyo VPS Products] All Tokyo VPS will come to End-of-Life on October 1st, 2023. Please backup all your data. We will issue pro-rated refunds accordingly. We are planning to launch new Tokyo-based services on our new site new.apernet.io.

[Postmortem : San Jose Service Outage Investigation] We have determined the previous outage was likely caused by a power surge, evidenced by the following incidents that happened simultaneously during the outage. 1. Dedicated server in off state was powered on with no human intervention. 2. Power supply error reported in some servers’ BMC log. 3. MikroTik switch failure - the switch failed to forward packets in hardware accelerated mode. Certain ports flapped constantly. A reboot has fixed the issues. 4. A GRUB bug in Debian related to LVM was triggered - an incorrect LVM volume ID was used in the boot loader. Manually adjusting the LVM volume size and rebuilding GRUB with the correct lvmid has fixed the issue. 5. The ZFS RAID array of a host server was down. A reboot and array rebuild has fixed the issue.

[Service Impacting: Hong Kong Provider Edge Switch Emergency Reboot] NOC noticed one of our provider edge switches was not working properly at 1430 HKT and issued an emergency reboot. This incident caused part of our upstream connections in Hong Kong to be down which caused some routing issues. The reboot is already done and the service should be back to normal. We apologize for any inconvenience you may experience during this incident. We thank you for your long-term trust and support.

[Service Outage : San Jose PVE Service on EPYC 7K62] The RAID array of the host server is down. There will be no data loss as we have all the backups. We are still investigating and trying to rebuild the RAID array.

[INCIDENT UPDATE: DDoS Attack on HK Services] NOC observed DDoS attack on HK services continues while we have mitigated some attack sent to our network. We seem to be only a tiny part of the widespread victims affected by today's large-scale DDoS attack. We have observed attacks with a peak bandwidth of more than 110Gbps. The attack may also be influencing other internet participants, which may also cause additional latency and packet loss to your usage. We will keep monitoring the situation. We apologize for any inconvenience you may experience during this attack. We thank you for your long-term trust and support.

[INCIDENT UPDATE: DDoS Attack on HK Services] NOC observed DDoS attack on HK services after router was rebooted, which influenced the network performance and caused packet loss. We are working with upstream provider to mitigate the attack. We apologize for any inconvenience you may experience during this incident. We thank you for your long-term trust and support.

[Service outage: Hong Kong] We observed that IP forwarding on one of our core routers is not running stably at 1300 HKT today, causing a service outage. An emergency reboot has been issued, and the service will gradually recover. We expect a 10-minute interruption for all HK services. We apologize for any inconvenience you may experience during this incident. We thank you for your long-term trust and support.

[INCIDENT UPDATE: Service Impacting Hong Kong Core Router Reboot] NOC investigated the incident and found the RAM usage of FPC is much higher than usual. We have planned a system reboot for the router on Sep 18, 0500 HKT to take mitigation resorts. Further action might also be performed in case of needed. We expect a 10-minute interruption for all HK services. We apologize for any inconvenience you may experience during this incident. We thank you for your long-term trust and support.

[maintenance: AperNet Hong Kong Core Router Emergency Reboot] Our core router at Hong Kong might be corrupted. A reboot is required to fix this issue. Please expect approx. 10 minutes downtime.