fa
Feedback
/var/log/DMIT-NOC.log

/var/log/DMIT-NOC.log

رفتن به کانال در Telegram
5 678
مشترکین
+524 ساعت
+947 روز
+40530 روز
آرشیو پست ها
LAX: ✅ The ARIN transfer has been done last week. ✅ The IPv4 has been announced. ✅ Partial routing is ready. 🚧 The first phase of the Los Angeles capacity expansion has been partially completed, reaching a deliverable state. 🚧 The second phase of upscale is ongoing. 🚧 Major network upgrade is in progress. 🚧 New subnet procurement is currently in progress. Pending: 🚧 Waiting for China Telecom finish filter update. DMIT already have ETA from CT. 🚧 Waiting for 1 more T1 IP Transit finish connection. (Non-blocking task) ============ APAC: ✅ The IPv4 for HKG and TYO has been announced. ✅ Partial routing is ready. 🚧 New network edge is 90% completed. It will be done within July. DMIT will use backbone to carry hot spot network to bypass the intra-asia congestion during peak. 🚧 HKG upgrade is ~95% completed, and reach deliverable state. Pending: 🚧 Waiting for China Telecom finish CN2 filter updated. DMIT already have ETA from CT. 🚧 Waiting for 2 more T1 IP Transit finish connection. (Non-blocking task)

We are still waiting ARIN to finish the transfer process. The hardware are on rack, waiting datacenters cross-connection. It'
We are still waiting ARIN to finish the transfer process. The hardware are on rack, waiting datacenters cross-connection. It's closed to finish the expansion.

Please be aware recently major web shell security breaches. https://github.com/nezhahq/nezha/security/advisories/GHSA-5c25-7vpj-9mqh DMIT will actively suspend VM if we received abuse notifications from other networks. This security breach gives shell access of your VM to public. This is prohibited by DMIT’s AUP. Based on the TOS and AUP, client has obligations to maintain the safety of their system and software. Due to the heavy support workload, once it was suspended, your VM might need more than few days to get unsuspended.

DMIT’s current strategy is to absorb the cost pressure associated with new infrastructure through direct acquisition, longer cost-allocation periods, and longer capital recovery cycles, while seeking to maintain stability in the renewal structure of existing products. Although the current procurement costs of servers, storage systems, and IP resources are significantly higher than in previous years, these assets can remain in service for multiple years. DMIT can therefore accept slower capital recovery and lower short-term margins rather than concentrating the cost of new infrastructure purchases into existing customer pricing. Based on DMIT’s recent quotations for newly procured resources and the adjustment mechanisms under existing contracts: - Memory and storage equipment prices have risen to approximately 2 to 2.5 times their 2025 levels. In some cases, refurbished DDR4-platform hardware now costs more than the brand-new equipment DMIT purchased three years ago. - Pricing for new datacenter orders and additional capacity has generally increased by approximately 30% to 50%, with some locations approaching twice their previous pricing. - DMIT’s existing datacenter contracts generally continue under their original contracted pricing, subject to the annual CPI adjustments specified in those agreements. Cross-connects, Internet Exchange ports, and other ancillary services are adjusted according to their respective contract terms. - Quotations for certain new local transport facilities have increased by approximately 80%, and some carriers have begun passing government-imposed fees through on new orders or during contractual adjustments. DMIT will continue using long-term procurement, resource consolidation, and longer capital recovery periods to reduce the impact of new infrastructure costs on the renewal structure of existing services. 4. Customer Support Restructuring DMIT remains committed to providing high-quality customer service and to offering competitive compensation and stable career development opportunities across our support, datacenter, development, and network teams. Due to the recent combination of cost and operational pressures, DMIT has slowed the pace of team expansion to avoid overextending company cash flow and compromising long-term employee stability. DMIT has not implemented proactive layoffs as a result of these pressures. DMIT previously operated multiple segmented support workflows. In practice, requests often required unnecessary handoffs, increasing workflow complexity and reducing the efficiency of information sharing, technical escalation, and incident handling. DMIT is therefore consolidating all customer support operations into a unified global support organization. Going forward, all customer support, ticket communication, and technical escalation will be conducted in English. 5. Conclusion DMIT is currently completing additional IP resource procurement, LAX infrastructure deployment, storage expansion, and the restructuring of its global support operations. These projects have required substantial capital, technical resources, and management attention, and have contributed to a temporary decline in certain aspects of service and customer support quality. We sincerely apologize for the inconvenience this has caused. DMIT’s immediate priorities are to progressively restore IP replacement processing and product sales, complete the LAX deployment, and finalize the new global support structure. DMIT is also deploying multiple new datacenter locations to support a wider range of customer requirements, including geographic coverage, network routing, latency, hardware configurations, and product types. DMIT will continue using long-term capital investment, multi-year cost allocation, and disciplined operational planning to maintain service quality, long-term sustainability, and stability in the renewal structure of existing products.

To All DMIT Customers and Partners: Service and Operational Update DMIT is currently working on several major initiatives in parallel, including IP resource procurement, LAX infrastructure expansion, storage deployment, and customer support restructuring. 1. IP Resources and LAX Progress IP Resources The relevant ARIN application has been formally approved, and we are completing the final provisioning and delivery process. LAX Hardware All servers have been racked and are awaiting completion of the datacenter’s power provisioning. The storage equipment has also arrived and is awaiting installation and configuration. Expected Sequence - Once the IP resources are ready, DMIT will resume processing IP replacement requests and announce the related compensation plan separately. - Once the servers are powered on and configured, and the IP resources are ready, DMIT will resume sales of selected products. - Once the storage expansion is complete, DMIT will fully resume product sales. The extended suspension of product sales and IP replacement processing was not caused by a single issue. While responding to recent special network incidents, DMIT also had to proceed with IP procurement and approval, LAX hardware deployment, and storage expansion. IP resources require a lengthy application, approval, and provisioning process. Running several infrastructure projects in parallel also created significant short-term capital requirements, engineering workloads, and operational pressure. To avoid increasing delivery obligations before sufficient IP, compute, and storage resources became available, DMIT temporarily suspended selected product sales and new IP replacement processing. These services will resume progressively according to the sequence above. 2. IP Replacement Policy and 95th-Percentile Billing IP Replacement and the TOS DMIT’s current IP leasing prices have risen to approximately 2.3 times their previous level. The IP replacement interval stated in the TOS represents the minimum eligibility period before a customer may submit another replacement request. It is not a guarantee that an IP address will automatically be replaced at each stated interval. Actual processing remains subject to IP availability, technical conditions, network operations, and abuse-prevention requirements. Historically, DMIT obtained a substantial portion of its IP resources through monthly leasing. As monthly leasing costs continue to rise, DMIT is gradually transitioning part of its IP inventory to direct acquisition. Direct acquisition requires greater upfront investment, but the cost can be allocated over an approximately five-year period. This converts an escalating monthly expense into a more stable and predictable long-term cost and reduces the potential impact on existing customer pricing. Transfer Quota Reset Policy All of DMIT’s upstream bandwidth is billed using the 95th-percentile, commonly referred to as 95/5, billing model. The traffic allowance and port speed of each service plan are designed around expected bandwidth costs during a billing period. Frequent traffic quota resets can cause actual usage to substantially exceed the plan’s original cost model and place additional pressure on network resources. DMIT therefore needs to adjust the pricing and usage rules for traffic quota resets to preserve fairness and ensure long-term service sustainability. Transfer quota reset is an additional, non-standard feature. It is not presented as a default service entitlement in the cart and is not guaranteed under the SLA or TOS. The adjustments made over the past several months were intended to evaluate actual usage patterns and support a more reasonable and sustainable long-term policy. 3. Responding to Industry Cost Increases Recurring expenses—including power, transportation/fiber, IP resources, and datacenter services—remain major drivers of operating cost increases. At the same time, the cost of newly procured servers, storage systems, IP resources, and additional datacenter capacity has risen significantly.

LAX: From CMI NOC: From 0:00 to 6:00 AM Hong Kong time on June 16th, NCP submarine cable will have a routine maintenance, which may affect IP transit service.

Update: 1. HKG.Pro plan (except existing one) has been upgraded to HKG.AN5.Pro The product will be available to order very soon. HKG AN5 is using DDR5 5600 9655P hardware. 2. Half of maintenance has been completed. The rest will be completed soon.

Scheduled Maintenance Location: Hong Kong Data Center Scope: All machines Maximum simultaneous impact unit: single hypervisor Maximum impact time: each VM restarts up to 2 times, 10 minutes each. Date: June 10, 2026 - June 17, 2026 Content: Data center migration and capacity expansion. Pro (except special discount) will be upgraded to AN5 platform during this migration.

LAX: AS9929 IPv6 inbound is ready for LAX.Pro and LAX.EB

To all customers: DMIT resources are in a very critical status. That's the reason we pause all sales at this time to minimize the impact on existing customers. We do apologize that there isn't any notice about the sales pause for weeks. For all locations: Our IPv4 address purchasing is under contract, we are getting more IPs very soon. We are pushing it to move fast but we cannot guarantee when it will be ready. Also once the IPv4 is transferred, we will publish it into the resource pool to release IP demand pressure and restore sales immediately. This might bring some routing inconsistency (main for inbound.) and geolocation issues (short time advertisement, and IP database update delay). We used to hold IP for more than 3-6 mo before it was put into the resource pool but this time we couldn't. For LAX, The hardware arrived last week, due to limited space, we are working on getting the new space and power. It takes a few more weeks. Even after it's ready, we won't restore sales for large scale instances. Because we need to scale up our storage cluster; the storage cluster is on the way to datacenter and it might take a little more times more once the computing node is ready. For HKG, Hardware is onsite, we are going to build it ASAP. Once the IP resource is ready, the sales will be restored. For TYO, We are only waiting for more IPs.

AS9929 states, the packet loss and jitter caused by submarine fiber cut between Shanghai-Tokyo segment. They states (not us) it has been fixed now. Let's see how it goes.

Again: AS9929 has fragile backbone, China Unicom is reconfiguring their network recently. DMIT has report this issue since 2 days ago when it actually happened. There is no useful information and no action has been done yet. China Unicom has never change, they always like this. AS58807: CMIN2 is heavily oversell, they cannot delivery full of purchased burst bandwidth. They don't have upgrade plan in any place. To improve overall experience, DMIT will have to divert some CM traffic to CU 9929 to reduce the load for CMIN2 after AS9929 back to normal. Based on test, when the AS9929 is stable, the path of AS9808-AS4837-AS9929 will be stable and fast. There is no evidence shows cross-ISP rate limit.

Management Facility Infrastructure Maintenance DMIT.io is hosted on an independent cloud infrastructure, which is currently undergoing scheduled maintenance. Estimated maintenance duration: 1 hour.

LAX.EB AS9929: We've already report fault case with China Unicom about AS9929 issue. DMIT has enough bandwidth with China Unicom. The bandwidth expansion has been completed but DMIT will not disclosure peering capacity any more in the future. DMIT still maintains a 50% bandwidth reserve. The current packet loss and fluctuations in latency are tentatively attributed to issues with China Unicom's backbone network. AS58807: CMIN2 is heavily oversold (On China Mobile US-CN backbone end). It has reach the ceiling of their capacity. However, they still sell small amounts of bandwidth to small customers. DMIT still maintains a 20% bandwidth reserve.

Some VM restarted outside of the scheduled maintenance window. This was caused by a Kernel Panic. Following the configuration corrections and kernel component updates, a final restart will still be required during the scheduled maintenance window. We sincerely apologize for any inconvenience this may cause.

The time zone has been changed.

Scheduled Maintenance Facility: Los Angeles Data Center Scope: All Virtual Machines Maximum Simultaneous Impact: One Hypervisor at a time Maximum Impact Duration: 1 hour per client VM Time: 3:00 PM – 5:00 PM UTC Dates: April 8 – April 16 Details: Resolving anomalies caused by suboptimal kernel configurations; a system reboot is required. Note: Only one Hypervisor will be processed at a time; therefore, the overall maintenance period will be extended.

LAX EB We’ve notice the jitter issue during the peak hours. DMIT is working with upstream and CUG to resolve this matter. It should be resolved sooner. TYO PRO We’ve switched major upstream of TYO Pro to Cloudflare. There is one prefixe has issue on CTGnet routing. We’ve already reported it to CTGnet.

HKG Pro: We are currently adjusting internet routing. Some ISPs have excessively long routing convergence times, leading to detours and blackouts. Thank you for your patience.