Computer Science and Programming
Channel specialized for advanced topics of: * Artificial intelligence, * Machine Learning, * Deep Learning, * Computer Vision, * Data Science * Python For Ads: @otchebuch & @cobbl, https://telega.io/c/computer_science_and_programming
نمایش بیشتر155 941مشترکین
+11524 ساعت
+1677 روز
+78430 روز
- مشترکین
- پوشش پست
- ER - نسبت تعامل
در حال بارگیری داده...
معدل نمو المشتركين
در حال بارگیری داده...
¡Hola! 👋🏻
AmigoChat - AI GPT bot. Best friend and assistant:
- generate images
- get ideas and hashtags for social media
- write SEO texts
- rewrite and summarize longreads
- choose a promotion plan
- chat and ask questions
Everything is FREE because amigos don't take dineros for help! 🤠
AmigoChat
AI GPT bot, friend, genius and good compañero
👍 13
So, how they solved it? 𝗧𝗵𝗲𝘆 𝗳𝗶𝗿𝘀𝘁 𝘁𝗿𝗶𝗲𝗱 𝘁𝗼 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱 𝗵𝗼𝘄 𝘁𝗵𝗲 𝘀𝘆𝘀𝘁𝗲𝗺 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝘀. They tracked what Elixir processes were doing, if they were stuck waiting on something, etc. They recorded the event types, how many of each kind of message they received, and their processing times. In addition, they tried to understand how much memory they use, the performances of garbage collectors, etc.
After the analysis, they 𝗰𝗿𝗲𝗮𝘁𝗲𝗱 𝘁𝗵𝗲 𝗳𝗼𝗹𝗹𝗼𝘄𝗶𝗻𝗴 𝘀𝘁𝗿𝗮𝘁𝗲𝗴𝘆:
𝟭. 𝗣𝗮𝘀𝘀𝗶𝘃𝗲 𝘀𝗲𝘀𝘀𝗶𝗼𝗻𝘀: Discord significantly reduced the amount of data processed and sent by differentiating between active and passive user connections, cutting the fanout work by 90% for large servers.
𝟮. 𝗥𝗲𝗹𝗮𝘆𝘀: Implementing a relay system (read - multithreading) allowed Discord to split the fanout process across multiple machines, enabling a single guild to utilize more resources and support more prominent communities. Relays maintain connections to the sessions instead of the guild and are responsible for doing fanout with permission checks.
𝟯. 𝗪𝗼𝗿𝗸𝗲𝗿 𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗲𝘀 𝗮𝗻𝗱 𝗘𝗧𝗦: To maintain server responsiveness, Discord employed worker processes and Erlang Term Storage (ETS) for operations requiring iteration over large sets of members, thus avoiding bottlenecks in the guild process. ETS is an in-memory database that supports the ability of multiple Elixir processes to access it safely. This enables the creation of a new worker process and passes the ETS table so this process can run expensive operations and offload the central guild server.
🔗https://discord.com/blog/maxjourney-pushing-discords-limits-with-a-million-plus-online-users-in-a-single-server
👍 18
𝗛𝗼𝘄 𝗱𝗼𝗲𝘀 𝗗𝗶𝘀𝗰𝗼𝗿𝗱 𝗵𝗮𝗻𝗱𝗹𝗲 𝗮 𝗺𝗶𝗹𝗹𝗶𝗼𝗻 𝗼𝗻𝗹𝗶𝗻𝗲 𝘂𝘀𝗲𝗿𝘀 𝗶𝗻 𝗮 𝘀𝗶𝗻𝗴𝗹𝗲 𝘀𝗲𝗿𝘃𝗲𝗿?
As time passed, the overall size of Discord's user base, including its most prominent communities, has grown massively. This affected servers that started to slow down and hit their throughput limits. So, they needed to scale individual Discord servers from tens of thousands to millions of concurrent users.
Whenever someone sends a message on Discord or joins a channel, they need to update the date UI of everyone online on that server. They call that server a "𝗴𝘂𝗶𝗹𝗱," which runs in a 𝘀𝗶𝗻𝗴𝗹𝗲 𝗘𝗹𝗶𝘅𝗶𝗿 𝗽𝗿𝗼𝗰𝗲𝘀𝘀, while there is another process (a "𝘀𝗲𝘀𝘀𝗶𝗼𝗻") for each connected client. The guild process tracks sessions of users who are members of that guild and are responsible for actions to those sessions. When sessions get updates, forward them to the web socket socket to the client.
The main issue is that 𝗮 𝘀𝗶𝗻𝗴𝗹𝗲 𝗺𝗲𝘀𝘀𝗮𝗴𝗲 𝗻𝗲𝗲𝗱𝘀 𝘁𝗼 𝗴𝗼 𝘁𝗼 𝘁𝗵𝗲 𝗻𝘂𝗺𝗯𝗲𝗿 𝗼𝗳 𝗽𝗲𝗼𝗽𝗹𝗲 𝗼𝗻𝗹𝗶𝗻𝗲 on that server, which means if a server has 1000 people online and they all send a message once, that's 1 million notifications.
👍 5👎 1
𝟮𝟬 𝗦𝗤𝗟 𝗾𝘂𝗲𝗿𝘆 𝗼𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝘁𝗲𝗰𝗵𝗻𝗶𝗾𝘂𝗲𝘀
Below are the SQL query optimization techniques that I found to be significant, listed in the top 20:
1. Create an index on huge tables (>1.000.000) rows
2. Use EXIST() instead of COUNT() to find an element in the table
3. SELECT fields instead of using SELECT *
4. Avoid Subqueries in WHERE Clause
5. Avoid SELECT DISTINCT where possible
6. Use WHERE Clause instead of HAVING
7. Create joins with INNER JOIN (not WHERE)
8. Use LIMIT to sample query results
9. Use UNION ALL instead of UNION wherever possible
10. Use UNION where instead of WHERE ... or ... query.
11. Run your query during off-peak hours
12. Avoid using OR in join queries
14. Choose GROUP BY over window functions
15. Use derived and temporary tables
16. Drop the index before loading bulk data
16. Use materialized views instead of views
17. Avoid != or <> (not equal) operator
18. Minimize the number of subqueries
19. Use INNER join as little as possible when you can get the same output using LEFT/RIGHT join.
20. Frequently try to use temporary sources to retrieve the same dataset.
👍 45
⏰Only a few hours left to join the Solcraft presale
Crypto markets are rebounding quickly and Bitcoin halving is in less than 5 days 🚀
Combine the most successful game title (Minecraft) with the fastest growing blockchain (Solana) and you create the hottest GameFi project of 2024 • Solcraft
✅Custom Minecraft Server - Live
✅$SOFT Token Integration Complete
✅Team with Ex-Microsoft & Steam Developers
Huge Partnerships 🤝 Massive Launch April 16th
The $SOFT presale ends in under 24 hours 👇
Presale
Find out more: Website • TG • Twitter
👍 11👎 1
𝗦𝘁𝗮𝗰𝗸 𝗢𝘃𝗲𝗿𝗳𝗹𝗼𝘄 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 𝗜𝘀 𝗡𝗼𝘁 𝗪𝗵𝗮𝘁 𝗬𝗼𝘂 𝗠𝗲𝗮𝗻 𝗜𝘁 𝗜𝘀
In the recent interview with Scott Hanselman, 𝗥𝗼𝗯𝗲𝗿𝘁𝗮 𝗔𝗿𝗰𝗼𝘃𝗲𝗿𝗱𝗲, 𝗛𝗲𝗮𝗱 𝗢𝗳 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗮𝘁 𝗦𝘁𝗮𝗰𝗸 𝗢𝘃𝗲𝗿𝗳𝗹𝗼𝘄, revealed the story about the architecture of Stack Overflow. They handle more than 6000 requests per second, 2 billion page views per month, and they manage to render a page in about 12 milliseconds. If we think about it a bit, we could imagine they use some kind of 𝗺𝗶𝗰𝗿𝗼𝘀𝗲𝗿𝘃𝗶𝗰𝗲 𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻 𝘁𝗵𝗮𝘁 𝗿𝘂𝗻𝘀 𝗶𝗻 𝘁𝗵𝗲 𝗖𝗹𝗼𝘂𝗱 𝘄𝗶𝘁𝗵 𝗞𝘂𝗯𝗲𝗿𝗻𝗲𝘁𝗲𝘀.
But the story is a bit different. Their solution is 15 years old, and it is a 𝗯𝗶𝗴 𝗺𝗼𝗻𝗼𝗹𝗶𝘁𝗵𝗶𝗰 𝗮𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻 𝗿𝘂𝗻𝗻𝗶𝗻𝗴 𝗼𝗻-𝗽𝗿𝗲𝗺𝗶𝘀𝗲𝘀. It is actually 𝗮 𝘀𝗶𝗻𝗴𝗹𝗲 𝗮𝗽𝗽 on IIS, which runs 200 sites. This single app is running on nine web servers and a single SQL Server (with the addition of one hot standby).
They also use 𝘁𝘄𝗼 𝗹𝗲𝘃𝗲𝗹𝘀 𝗼𝗳 𝗰𝗮𝗰𝗵𝗲, one on SQL Server with large RAM (1.5TB), where they have 30% of DB access in RAM and also they use two Redis servers (master and replica). Besides this, they have 3 tag engine servers and 3 Elastic search servers, which are used for 34 million daily searches.
All this is handled by a 𝘁𝗲𝗮𝗺 𝗼𝗳 𝟱𝟬 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝘀, who manage to 𝗱𝗲𝗽𝗹𝗼𝘆 𝘁𝗼 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝗶𝗻 𝟰 𝗺𝗶𝗻𝘀 several times daily.
Their 𝗳𝘂𝗹𝗹 𝘁𝗲𝗰𝗵 𝘀𝘁𝗮𝗰𝗸 is:
🔹 C# + ASP. NET MVC
🔹 Dapper ORM
🔹 StaeckExchange Redis
🔹 MiniProfiler
🔹 Jil JSON Deseliazier
🔹 Exceptional logger for SQL
🔹 Sigil, a .Net CIL generation helper (for when C# isn’t fast enough)
🔹 NetGain, a high-performance web socket server
🔹 Opserver, monitoring dashboard polling most systems and feeding from Orion, Bosun, or WMI.
🔹 Bosun, backend monitoring system, written in Go
👍 44
𝗛𝗼𝘄 𝗧𝗼 𝗘𝗻𝗮𝗯𝗹𝗲 𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻 𝘄𝗶𝘁𝗵 𝗣𝘂𝗹𝗹 𝗥𝗲𝗾𝘂𝗲𝘀𝘁𝘀?
With Pull Requests, we lost the ability to have a proper Continuous Integration (CI) process in a way that delayed integration due to code reviews. So here comes a “Ship/Show/Ask” branching strategy. The thing is that not all pull requests need code reviews.
So, whenever we make a change, we have three options:
🔹 𝗦𝗵𝗶𝗽 - Small changes that don’t need people’s review can be pushed directly to the main branch. We have some build pipelines running on the main brunch, which run tests and other checks, so it is a safety net for our changes. Some examples are: fixing a typo, increasing the minor dependency version, updated documentation.
🔹 𝗦𝗵𝗼𝘄 - Here, we want to show what has been done. When you have a branch, you open a Pull Request and merge it without a review. Yet, you still want people to be notified of the change (to review it later), but don’t expect essential discussions. Some examples are: a local refactoring, fixing a bug, added a test case.
🔹 𝗔𝘀𝗸 - Here, we make our changes and open a Pull Request while waiting for feedback. We do this because we want a proper review in case we need clarification on our approach. This is a classical way of making Pull Requests. Some examples are: Adding a new feature, major refactoring, and proof of concept.
👍 35