en
Feedback
/var/log/DMIT-NOC.log

/var/log/DMIT-NOC.log

Open in Telegram
5 930
Subscribers
+724 hours
+577 days
+34430 days
Posts Archive
DMIT has switched all traffic to non faulty PoP. The network should be stable. We will revert back to redundancy PoP connectivity once CT resolved the issue.

Incident notice Location: Los Angeles Data Center Network Profile: Premium DMIT has identify there is some issue on China Telecom CN2 backbone caused on going packet loss on LAX.Pro We've reached out China Telecom about this matter.

Maintenance Notice Region: Los Angeles Data Center Scope: AN4/AN5 hypervisor Impact: Approximately 5 minutes (maximum 30 minutes); Only one hypervisor each time. Dates: July 16, 2026 – July 31, 2026 (UTC) Reason: New data center facility has been delivered; The existing data center has design defects on power infrastructure; migrating of all existing infrastructure to a higher-standard data center.

Maintenance has been completed. All switches have the most updated JunOS firmware installed. So far it's stable and running well. We will keep monitor it.

It will be done soon. There are multiple switches in-line needs to do the same upgrade.

Normally, performing maintenance on one switch would not affect the other. However, this JunOS firmware version contains a critical bug: protocol renegotiation triggers a protocol restart or even a kernel reboot. When we were updating one of the switches, the other one rebooted unexpectedly. That bring network down for 3 mins. This is the issue we are attempting to resolve with this update.

Emergency Maintenance Location: LAX New Location Details: Fixing a switch firmware error Impact: LAX.AS3

A free, unconditional self-service IPv4 address change opportunity has been issued. This free opportunity can be used before September 1, 2026, and applies to all product services created before July 1, 2026. It is limited to one use per service, with no additional restrictive conditions and no manual review required. Please note that once an IP address is changed, it cannot be reverted. Furthermore, DMIT does not guarantee the accessibility of the IP address on any third-party websites, streaming platforms, or AI services, nor does it guarantee the geographic location information of the IP address on specific testing platforms.

Security Maintenance Notification for past emergency maintenance done by APAC Team. Location: Los Angeles and Hong Kong Datacenter Scope: Part of LAX and HKG KVM virtualization nodes. Impact: Some customer VMs may have experienced a short restart or temporary service interruption Date: 2026-07-07 UTC Reason: Mitigation for Januscape, CVE-2026-53359 DMIT has restarted some services and selected virtualization components to mitigate the recently disclosed Januscape vulnerability. Januscape affects KVM/x86 virtualization isolation. To reduce exposure and maintain platform safety, we performed controlled restarts on affected service components in LAX and HKG. The operation was completed in batches to minimize simultaneous customer impact. Current status: • Required service restarts have been completed. • Affected services are being monitored. • Additional kernel or virtualization component updates may still require future maintenance windows. • No customer action is required at this time unless separately notified. We apologize for any inconvenience caused by the short service interruption. DMIT will continue to monitor the situation and apply further updates if required.

LAX: ✅ The ARIN transfer has been done last week. ✅ The IPv4 has been announced. ✅ Partial routing is ready. 🚧 The first phase of the Los Angeles capacity expansion has been partially completed, reaching a deliverable state. 🚧 The second phase of upscale is ongoing. 🚧 Major network upgrade is in progress. 🚧 New subnet procurement is currently in progress. Pending: 🚧 Waiting for China Telecom finish filter update. DMIT already have ETA from CT. 🚧 Waiting for 1 more T1 IP Transit finish connection. (Non-blocking task) ============ APAC: ✅ The IPv4 for HKG and TYO has been announced. ✅ Partial routing is ready. 🚧 New network edge is 90% completed. It will be done within July. DMIT will use backbone to carry hot spot network to bypass the intra-asia congestion during peak. 🚧 HKG upgrade is ~95% completed, and reach deliverable state. Pending: 🚧 Waiting for China Telecom finish CN2 filter updated. DMIT already have ETA from CT. 🚧 Waiting for 2 more T1 IP Transit finish connection. (Non-blocking task)

We are still waiting ARIN to finish the transfer process. The hardware are on rack, waiting datacenters cross-connection. It'
We are still waiting ARIN to finish the transfer process. The hardware are on rack, waiting datacenters cross-connection. It's closed to finish the expansion.

Please be aware recently major web shell security breaches. https://github.com/nezhahq/nezha/security/advisories/GHSA-5c25-7vpj-9mqh DMIT will actively suspend VM if we received abuse notifications from other networks. This security breach gives shell access of your VM to public. This is prohibited by DMIT’s AUP. Based on the TOS and AUP, client has obligations to maintain the safety of their system and software. Due to the heavy support workload, once it was suspended, your VM might need more than few days to get unsuspended.

DMIT’s current strategy is to absorb the cost pressure associated with new infrastructure through direct acquisition, longer cost-allocation periods, and longer capital recovery cycles, while seeking to maintain stability in the renewal structure of existing products. Although the current procurement costs of servers, storage systems, and IP resources are significantly higher than in previous years, these assets can remain in service for multiple years. DMIT can therefore accept slower capital recovery and lower short-term margins rather than concentrating the cost of new infrastructure purchases into existing customer pricing. Based on DMIT’s recent quotations for newly procured resources and the adjustment mechanisms under existing contracts: - Memory and storage equipment prices have risen to approximately 2 to 2.5 times their 2025 levels. In some cases, refurbished DDR4-platform hardware now costs more than the brand-new equipment DMIT purchased three years ago. - Pricing for new datacenter orders and additional capacity has generally increased by approximately 30% to 50%, with some locations approaching twice their previous pricing. - DMIT’s existing datacenter contracts generally continue under their original contracted pricing, subject to the annual CPI adjustments specified in those agreements. Cross-connects, Internet Exchange ports, and other ancillary services are adjusted according to their respective contract terms. - Quotations for certain new local transport facilities have increased by approximately 80%, and some carriers have begun passing government-imposed fees through on new orders or during contractual adjustments. DMIT will continue using long-term procurement, resource consolidation, and longer capital recovery periods to reduce the impact of new infrastructure costs on the renewal structure of existing services. 4. Customer Support Restructuring DMIT remains committed to providing high-quality customer service and to offering competitive compensation and stable career development opportunities across our support, datacenter, development, and network teams. Due to the recent combination of cost and operational pressures, DMIT has slowed the pace of team expansion to avoid overextending company cash flow and compromising long-term employee stability. DMIT has not implemented proactive layoffs as a result of these pressures. DMIT previously operated multiple segmented support workflows. In practice, requests often required unnecessary handoffs, increasing workflow complexity and reducing the efficiency of information sharing, technical escalation, and incident handling. DMIT is therefore consolidating all customer support operations into a unified global support organization. Going forward, all customer support, ticket communication, and technical escalation will be conducted in English. 5. Conclusion DMIT is currently completing additional IP resource procurement, LAX infrastructure deployment, storage expansion, and the restructuring of its global support operations. These projects have required substantial capital, technical resources, and management attention, and have contributed to a temporary decline in certain aspects of service and customer support quality. We sincerely apologize for the inconvenience this has caused. DMIT’s immediate priorities are to progressively restore IP replacement processing and product sales, complete the LAX deployment, and finalize the new global support structure. DMIT is also deploying multiple new datacenter locations to support a wider range of customer requirements, including geographic coverage, network routing, latency, hardware configurations, and product types. DMIT will continue using long-term capital investment, multi-year cost allocation, and disciplined operational planning to maintain service quality, long-term sustainability, and stability in the renewal structure of existing products.

To All DMIT Customers and Partners: Service and Operational Update DMIT is currently working on several major initiatives in parallel, including IP resource procurement, LAX infrastructure expansion, storage deployment, and customer support restructuring. 1. IP Resources and LAX Progress IP Resources The relevant ARIN application has been formally approved, and we are completing the final provisioning and delivery process. LAX Hardware All servers have been racked and are awaiting completion of the datacenter’s power provisioning. The storage equipment has also arrived and is awaiting installation and configuration. Expected Sequence - Once the IP resources are ready, DMIT will resume processing IP replacement requests and announce the related compensation plan separately. - Once the servers are powered on and configured, and the IP resources are ready, DMIT will resume sales of selected products. - Once the storage expansion is complete, DMIT will fully resume product sales. The extended suspension of product sales and IP replacement processing was not caused by a single issue. While responding to recent special network incidents, DMIT also had to proceed with IP procurement and approval, LAX hardware deployment, and storage expansion. IP resources require a lengthy application, approval, and provisioning process. Running several infrastructure projects in parallel also created significant short-term capital requirements, engineering workloads, and operational pressure. To avoid increasing delivery obligations before sufficient IP, compute, and storage resources became available, DMIT temporarily suspended selected product sales and new IP replacement processing. These services will resume progressively according to the sequence above. 2. IP Replacement Policy and 95th-Percentile Billing IP Replacement and the TOS DMIT’s current IP leasing prices have risen to approximately 2.3 times their previous level. The IP replacement interval stated in the TOS represents the minimum eligibility period before a customer may submit another replacement request. It is not a guarantee that an IP address will automatically be replaced at each stated interval. Actual processing remains subject to IP availability, technical conditions, network operations, and abuse-prevention requirements. Historically, DMIT obtained a substantial portion of its IP resources through monthly leasing. As monthly leasing costs continue to rise, DMIT is gradually transitioning part of its IP inventory to direct acquisition. Direct acquisition requires greater upfront investment, but the cost can be allocated over an approximately five-year period. This converts an escalating monthly expense into a more stable and predictable long-term cost and reduces the potential impact on existing customer pricing. Transfer Quota Reset Policy All of DMIT’s upstream bandwidth is billed using the 95th-percentile, commonly referred to as 95/5, billing model. The traffic allowance and port speed of each service plan are designed around expected bandwidth costs during a billing period. Frequent traffic quota resets can cause actual usage to substantially exceed the plan’s original cost model and place additional pressure on network resources. DMIT therefore needs to adjust the pricing and usage rules for traffic quota resets to preserve fairness and ensure long-term service sustainability. Transfer quota reset is an additional, non-standard feature. It is not presented as a default service entitlement in the cart and is not guaranteed under the SLA or TOS. The adjustments made over the past several months were intended to evaluate actual usage patterns and support a more reasonable and sustainable long-term policy. 3. Responding to Industry Cost Increases Recurring expenses—including power, transportation/fiber, IP resources, and datacenter services—remain major drivers of operating cost increases. At the same time, the cost of newly procured servers, storage systems, IP resources, and additional datacenter capacity has risen significantly.

LAX: From CMI NOC: From 0:00 to 6:00 AM Hong Kong time on June 16th, NCP submarine cable will have a routine maintenance, which may affect IP transit service.

Update: 1. HKG.Pro plan (except existing one) has been upgraded to HKG.AN5.Pro The product will be available to order very soon. HKG AN5 is using DDR5 5600 9655P hardware. 2. Half of maintenance has been completed. The rest will be completed soon.

Scheduled Maintenance Location: Hong Kong Data Center Scope: All machines Maximum simultaneous impact unit: single hypervisor Maximum impact time: each VM restarts up to 2 times, 10 minutes each. Date: June 10, 2026 - June 17, 2026 Content: Data center migration and capacity expansion. Pro (except special discount) will be upgraded to AN5 platform during this migration.

LAX: AS9929 IPv6 inbound is ready for LAX.Pro and LAX.EB