Computer Science and Programming
Channel specialized for advanced topics of: * Artificial intelligence, * Machine Learning, * Deep Learning, * Computer Vision, * Data Science * Python For Ads: @otchebuch & @cobbl, https://telega.io/c/computer_science_and_programming
Show more156 212Subscribers
+4924 hours
+2017 days
+51230 days
Posting time distributions
Data loading in progress...
Traffic sources
- Subscribers 0.00%
- Channels 0.00%
- Via the link 0.00%
- Groups 0.00%
- Telegram search 0.00%
- Direct messages 0.00%
- Other 0.00%
Publication analysis
Posts | Views | Shares | Views dynamics |
01 Challenging programming projects you should try:
๐https://jamesg.blog/2024/02/28/programming-projects/https://jamesg.blog/2024/02/28/programming-projects/ | 14 369 | 36 | Loading... |
02 Who's here?ย
We've asked for a free link to a paid channel, for our subs.
x2-x3 Signals here
๐ CLICK HERE TO JOIN ๐
๐ CLICK HERE TO JOIN ๐
๐ CLICK HERE TO JOIN ๐
โ๏ธJOIN FAST! FIRST 1000 SUBS WILL BE ACCEPTED | 13 971 | 3 | Loading... |
03 #promo | 11 381 | 0 | Loading... |
04 ๐๐ผ๐ ๐๐ผ ๐ฐ๐ผ๐ฑ๐ฒ ๐๐ถ๐๐ต ๐๐ถ๐๐๐๐ฏ ๐๐ผ๐ฝ๐ถ๐น๐ผ๐?
A recent study by GitHub and Microsoft discovered that AI now authors 46% of new code. They also found that overall developer productivity surged by 55%, leading to more efficient coding processes. When we talk about AI-powered coding, we mainly talk about GitHub Copilot.
But ๐ต๐ผ๐ ๐๐ถ๐๐๐๐ฏ ๐๐ผ๐ฝ๐ถ๐น๐ผ๐ ๐๐ผ๐ฟ๐ธ๐?
The process goes in the following steps:
๐ญ. ๐ฆ๐ฒ๐ฐ๐๐ฟ๐ฒ ๐ฝ๐ฟ๐ผ๐บ๐ฝ๐ ๐๐ฟ๐ฎ๐ป๐๐บ๐ถ๐๐๐ถ๐ผ๐ป: Your prompts are securely sent to Copilot, ensuring data privacy.
๐ฎ. ๐๐ผ๐ป๐๐ฒ๐
๐๐๐ฎ๐น ๐๐ป๐ฑ๐ฒ๐ฟ๐๐๐ฎ๐ป๐ฑ๐ถ๐ป๐ด: Copilot analyzes the code around your cursor, the file type, and other open files to offer relevant suggestions.
๐ฏ. ๐๐ผ๐ป๐๐ฒ๐ป๐ ๐ณ๐ถ๐น๐๐ฒ๐ฟ๐ถ๐ป๐ด: It filters out personal data and inappropriate content, focusing solely on generating helpful code.
๐ฐ. ๐๐ผ๐ฑ๐ฒ ๐ด๐ฒ๐ป๐ฒ๐ฟ๐ฎ๐๐ถ๐ผ๐ป: Based on the intent identified in your prompts, Copilot crafts code suggestions that align with your coding style and project standards.
๐ฑ. ๐จ๐๐ฒ๐ฟ ๐ถ๐ป๐๐ฒ๐ฟ๐ฎ๐ฐ๐๐ถ๐ผ๐ป: Here, we can decide whether to use, tweak, or reject Copilot's suggestions.
๐ฒ. ๐๐ฒ๐ฒ๐ฑ๐ฏ๐ฎ๐ฐ๐ธ ๐น๐ผ๐ผ๐ฝ: Copilot learns from your interactions, improving its suggestions. Every time you tweak or reject its ideas, he knows from it. It employs techniques like zero-shot (asking without examples), one-shot (asking with an example), and few-shot learning (providing multiple examples) to adapt to our instructions, whether you provide examples or not.
๐ณ. ๐ฃ๐ฟ๐ผ๐บ๐ฝ๐ ๐ต๐ถ๐๐๐ผ๐ฟ๐ ๐ฟ๐ฒ๐๐ฒ๐ป๐๐ถ๐ผ๐ป: It remembers past prompts and interactions, making future suggestions more accurate. | 12 536 | 30 | Loading... |
05 โ Message was hidden by channel owner | 1 | 0 | Loading... |
06 โ Message was hidden by channel owner | 9 682 | 22 | Loading... |
07 โ Message was hidden by channel owner | 9 306 | 6 | Loading... |
08 โ Message was hidden by channel owner | 8 750 | 1 | Loading... |
09 So, how they solved it? ๐ง๐ต๐ฒ๐ ๐ณ๐ถ๐ฟ๐๐ ๐๐ฟ๐ถ๐ฒ๐ฑ ๐๐ผ ๐๐ป๐ฑ๐ฒ๐ฟ๐๐๐ฎ๐ป๐ฑ ๐ต๐ผ๐ ๐๐ต๐ฒ ๐๐๐๐๐ฒ๐บ ๐ฝ๐ฒ๐ฟ๐ณ๐ผ๐ฟ๐บ๐. They tracked what Elixir processes were doing, if they were stuck waiting on something, etc. They recorded the event types, how many of each kind of message they received, and their processing times. In addition, they tried to understand how much memory they use, the performances of garbage collectors, etc.
After the analysis, they ๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ๐ฑ ๐๐ต๐ฒ ๐ณ๐ผ๐น๐น๐ผ๐๐ถ๐ป๐ด ๐๐๐ฟ๐ฎ๐๐ฒ๐ด๐:
๐ญ. ๐ฃ๐ฎ๐๐๐ถ๐๐ฒ ๐๐ฒ๐๐๐ถ๐ผ๐ป๐: Discord significantly reduced the amount of data processed and sent by differentiating between active and passive user connections, cutting the fanout work by 90% for large servers.
๐ฎ. ๐ฅ๐ฒ๐น๐ฎ๐๐: Implementing a relay system (read - multithreading) allowed Discord to split the fanout process across multiple machines, enabling a single guild to utilize more resources and support more prominent communities. Relays maintain connections to the sessions instead of the guild and are responsible for doing fanout with permission checks.
๐ฏ. ๐ช๐ผ๐ฟ๐ธ๐ฒ๐ฟ ๐ฝ๐ฟ๐ผ๐ฐ๐ฒ๐๐๐ฒ๐ ๐ฎ๐ป๐ฑ ๐๐ง๐ฆ: To maintain server responsiveness, Discord employed worker processes and Erlang Term Storage (ETS) for operations requiring iteration over large sets of members, thus avoiding bottlenecks in the guild process. ETS is an in-memory database that supports the ability of multiple Elixir processes to access it safely. This enables the creation of a new worker process and passes the ETS table so this process can run expensive operations and offload the central guild server.
๐https://discord.com/blog/maxjourney-pushing-discords-limits-with-a-million-plus-online-users-in-a-single-server | 15 031 | 27 | Loading... |
10 ๐๐ผ๐ ๐ฑ๐ผ๐ฒ๐ ๐๐ถ๐๐ฐ๐ผ๐ฟ๐ฑ ๐ต๐ฎ๐ป๐ฑ๐น๐ฒ ๐ฎ ๐บ๐ถ๐น๐น๐ถ๐ผ๐ป ๐ผ๐ป๐น๐ถ๐ป๐ฒ ๐๐๐ฒ๐ฟ๐ ๐ถ๐ป ๐ฎ ๐๐ถ๐ป๐ด๐น๐ฒ ๐๐ฒ๐ฟ๐๐ฒ๐ฟ?
As time passed, the overall size of Discord's user base, including its most prominent communities, has grown massively. This affected servers that started to slow down and hit their throughput limits. So, they needed to scale individual Discord servers from tens of thousands to millions of concurrent users.
Whenever someone sends a message on Discord or joins a channel, they need to update the date UI of everyone online on that server. They call that server a "๐ด๐๐ถ๐น๐ฑ," which runs in a ๐๐ถ๐ป๐ด๐น๐ฒ ๐๐น๐ถ๐
๐ถ๐ฟ ๐ฝ๐ฟ๐ผ๐ฐ๐ฒ๐๐, while there is another process (a "๐๐ฒ๐๐๐ถ๐ผ๐ป") for each connected client. The guild process tracks sessions of users who are members of that guild and are responsible for actions to those sessions. When sessions get updates, forward them to the web socket socket to the client.
The main issue is that ๐ฎ ๐๐ถ๐ป๐ด๐น๐ฒ ๐บ๐ฒ๐๐๐ฎ๐ด๐ฒ ๐ป๐ฒ๐ฒ๐ฑ๐ ๐๐ผ ๐ด๐ผ ๐๐ผ ๐๐ต๐ฒ ๐ป๐๐บ๐ฏ๐ฒ๐ฟ ๐ผ๐ณ ๐ฝ๐ฒ๐ผ๐ฝ๐น๐ฒ ๐ผ๐ป๐น๐ถ๐ป๐ฒ on that server, which means if a server has 1000 people online and they all send a message once, that's 1 million notifications. | 12 533 | 35 | Loading... |
11 ๐ฎ๐ฌ ๐ฆ๐ค๐ ๐พ๐๐ฒ๐ฟ๐ ๐ผ๐ฝ๐๐ถ๐บ๐ถ๐๐ฎ๐๐ถ๐ผ๐ป ๐๐ฒ๐ฐ๐ต๐ป๐ถ๐พ๐๐ฒ๐
Below are the SQL query optimization techniques that I found to be significant, listed in the top 20:
1. Create an index on huge tables (>1.000.000) rows
2. Use EXIST() instead of COUNT() to find an element in the table
3. SELECT fields instead of using SELECT *
4. Avoid Subqueries in WHERE Clause
5. Avoid SELECT DISTINCT where possible
6. Use WHERE Clause instead of HAVING
7. Create joins with INNER JOIN (not WHERE)
8. Use LIMIT to sample query results
9. Use UNION ALL instead of UNION wherever possible
10. Use UNION where instead of WHERE ... or ... query.
11. Run your query during off-peak hours
12. Avoid using OR in join queries
14. Choose GROUP BY over window functions
15. Use derived and temporary tables
16. Drop the index before loading bulk data
16. Use materialized views instead of views
17. Avoid != or <> (not equal) operator
18. Minimize the number of subqueries
19. Use INNER join as little as possible when you can get the same output using LEFT/RIGHT join.
20. Frequently try to use temporary sources to retrieve the same dataset. | 14 735 | 140 | Loading... |
12 โ Message was hidden by channel owner | 11 882 | 10 | Loading... |
13 โ Message was hidden by channel owner | 10 214 | 0 | Loading... |
14 ๐ฆ๐๐ฎ๐ฐ๐ธ ๐ข๐๐ฒ๐ฟ๐ณ๐น๐ผ๐ ๐๐ฟ๐ฐ๐ต๐ถ๐๐ฒ๐ฐ๐๐๐ฟ๐ฒ ๐๐ ๐ก๐ผ๐ ๐ช๐ต๐ฎ๐ ๐ฌ๐ผ๐ ๐ ๐ฒ๐ฎ๐ป ๐๐ ๐๐
In the recent interview with Scott Hanselman, ๐ฅ๐ผ๐ฏ๐ฒ๐ฟ๐๐ฎ ๐๐ฟ๐ฐ๐ผ๐๐ฒ๐ฟ๐ฑ๐ฒ, ๐๐ฒ๐ฎ๐ฑ ๐ข๐ณ ๐๐ป๐ด๐ถ๐ป๐ฒ๐ฒ๐ฟ๐ถ๐ป๐ด ๐ฎ๐ ๐ฆ๐๐ฎ๐ฐ๐ธ ๐ข๐๐ฒ๐ฟ๐ณ๐น๐ผ๐, revealed the story about the architecture of Stack Overflow. They handle more than 6000 requests per second, 2 billion page views per month, and they manage to render a page in about 12 milliseconds. If we think about it a bit, we could imagine they use some kind of ๐บ๐ถ๐ฐ๐ฟ๐ผ๐๐ฒ๐ฟ๐๐ถ๐ฐ๐ฒ ๐๐ผ๐น๐๐๐ถ๐ผ๐ป ๐๐ต๐ฎ๐ ๐ฟ๐๐ป๐ ๐ถ๐ป ๐๐ต๐ฒ ๐๐น๐ผ๐๐ฑ ๐๐ถ๐๐ต ๐๐๐ฏ๐ฒ๐ฟ๐ป๐ฒ๐๐ฒ๐.
But the story is a bit different. Their solution is 15 years old, and it is a ๐ฏ๐ถ๐ด ๐บ๐ผ๐ป๐ผ๐น๐ถ๐๐ต๐ถ๐ฐ ๐ฎ๐ฝ๐ฝ๐น๐ถ๐ฐ๐ฎ๐๐ถ๐ผ๐ป ๐ฟ๐๐ป๐ป๐ถ๐ป๐ด ๐ผ๐ป-๐ฝ๐ฟ๐ฒ๐บ๐ถ๐๐ฒ๐. It is actually ๐ฎ ๐๐ถ๐ป๐ด๐น๐ฒ ๐ฎ๐ฝ๐ฝ on IIS, which runs 200 sites. This single app is running on nine web servers and a single SQL Server (with the addition of one hot standby).
They also use ๐๐๐ผ ๐น๐ฒ๐๐ฒ๐น๐ ๐ผ๐ณ ๐ฐ๐ฎ๐ฐ๐ต๐ฒ, one on SQL Server with large RAM (1.5TB), where they have 30% of DB access in RAM and also they use two Redis servers (master and replica). Besides this, they have 3 tag engine servers and 3 Elastic search servers, which are used for 34 million daily searches.
All this is handled by a ๐๐ฒ๐ฎ๐บ ๐ผ๐ณ ๐ฑ๐ฌ ๐ฒ๐ป๐ด๐ถ๐ป๐ฒ๐ฒ๐ฟ๐, who manage to ๐ฑ๐ฒ๐ฝ๐น๐ผ๐ ๐๐ผ ๐ฝ๐ฟ๐ผ๐ฑ๐๐ฐ๐๐ถ๐ผ๐ป ๐ถ๐ป ๐ฐ ๐บ๐ถ๐ป๐ several times daily.
Their ๐ณ๐๐น๐น ๐๐ฒ๐ฐ๐ต ๐๐๐ฎ๐ฐ๐ธ is:
๐น C# + ASP. NET MVC
๐น Dapper ORM
๐น StaeckExchange Redis
๐น MiniProfiler
๐น Jil JSON Deseliazier
๐น Exceptional logger for SQL
๐น Sigil, a .Net CIL generation helper (for when C# isnโt fast enough)
๐น NetGain, a high-performance web socket server
๐น Opserver, monitoring dashboard polling most systems and feeding from Orion, Bosun, or WMI.
๐น Bosun, backend monitoring system, written in Go | 19 155 | 51 | Loading... |
15 ๐๐ผ๐ ๐ง๐ผ ๐๐ป๐ฎ๐ฏ๐น๐ฒ ๐๐ผ๐ป๐๐ถ๐ป๐๐ผ๐๐ ๐๐ป๐๐ฒ๐ด๐ฟ๐ฎ๐๐ถ๐ผ๐ป ๐๐ถ๐๐ต ๐ฃ๐๐น๐น ๐ฅ๐ฒ๐พ๐๐ฒ๐๐๐?
With Pull Requests, we lost the ability to have a proper Continuous Integration (CI) process in a way that delayed integration due to code reviews. So here comes a โShip/Show/Askโ branching strategy. The thing is that not all pull requests need code reviews.
So, whenever we make a change, we have three options:
๐น ๐ฆ๐ต๐ถ๐ฝ - Small changes that donโt need peopleโs review can be pushed directly to the main branch. We have some build pipelines running on the main brunch, which run tests and other checks, so it is a safety net for our changes. Some examples are: fixing a typo, increasing the minor dependency version, updated documentation.
๐น ๐ฆ๐ต๐ผ๐ - Here, we want to show what has been done. When you have a branch, you open a Pull Request and merge it without a review. Yet, you still want people to be notified of the change (to review it later), but donโt expect essential discussions. Some examples are: a local refactoring, fixing a bug, added a test case.
๐น ๐๐๐ธ - Here, we make our changes and open a Pull Request while waiting for feedback. We do this because we want a proper review in case we need clarification on our approach. This is a classical way of making Pull Requests. Some examples are: Adding a new feature, major refactoring, and proof of concept. | 24 814 | 29 | Loading... |
16 โ Message was hidden by channel owner | 21 284 | 13 | Loading... |
17 โ Message was hidden by channel owner | 20 205 | 1 | Loading... |
18 Encryption and Decryption using Linear Algebra with C++
This project implements a text encryption and decryption system using a matrix-based encryption technique. This project serves as an educational and practical exploration of matrix-based encryption techniques, demonstrating the fundamental concepts of encryption and decryption in a user-friendly manner.
๐ปhttps://github.com/farukalpay/TextEncryptionWithLinearAlgebra | 26 269 | 59 | Loading... |
19 ๐๐ผ๐ ๐๐ผ ๐ฑ๐ผ ๐ฐ๐ผ๐ฑ๐ฒ ๐ฟ๐ฒ๐๐ถ๐ฒ๐๐ ๐ฝ๐ฟ๐ผ๐ฝ๐ฒ๐ฟ๐น๐
An essential step in the software development lifecycle is code review. It enables developers to enhance code quality significantly. It resembles the authoring of a book. The author writes the story, which is then edited to ensure no mistakes like mixing up "you're" with "yours." Code review in this context refers to examining and assessing other people's code.
There are different ๐ฏ๐ฒ๐ป๐ฒ๐ณ๐ถ๐๐ ๐ผ๐ณ ๐ฎ ๐ฐ๐ผ๐ฑ๐ฒ ๐ฟ๐ฒ๐๐ถ๐ฒ๐: it ensures consistency in design and implementation, optimizes code for better performance, is an opportunity to learn, and knowledge sharing and mentoring, as well as promotes team cohesion.
What should you look for in a code review? Try to look for things such as:
๐น ๐๐ฒ๐๐ถ๐ด๐ป (does this integrate well with the rest of the system, and are interactions of different components make sense)
๐น ๐๐๐ป๐ฐ๐๐ถ๐ผ๐ป๐ฎ๐น๐ถ๐๐ (does this change is what the developer intended)
๐น ๐๐ผ๐บ๐ฝ๐น๐ฒ๐
๐ถ๐๐ (is this code more complex than it should be)
๐น ๐ก๐ฎ๐บ๐ถ๐ป๐ด (is naming good?)
๐น ๐๐ป๐ด. ๐ฝ๐ฟ๐ถ๐ป๐ฐ๐ถ๐ฝ๐น๐ฒ๐ (solid, kiss, dry)
๐น ๐ง๐ฒ๐๐๐ (are different kinds of tests used appropriately, code coverage),
๐น ๐ฆ๐๐๐น๐ฒ (does it follow style guidelines),
๐น ๐๐ผ๐ฐ๐๐บ๐ฒ๐ป๐๐ฎ๐๐ถ๐ผ๐ป, etc. | 34 113 | 72 | Loading... |
20 โ Message was hidden by channel owner | 7 873 | 1 | Loading... |
21 โ Message was hidden by channel owner | 7 302 | 0 | Loading... |
22 โ Message was hidden by channel owner | 7 840 | 0 | Loading... |
23 โ Message was hidden by channel owner | 9 397 | 7 | Loading... |
24 โ Message was hidden by channel owner | 14 365 | 3 | Loading... |
25 ๐๐ถ๐ฑ ๐ ๐ด๐ถ๐๐ฒ ๐บ๐ ๐ฏ๐ฒ๐๐ ๐น๐ฎ๐๐ ๐๐ฒ๐ฒ๐ธ?
There are no two same days nor two same weeks
The "best" can mean different on "different" days
This is why we need to have weekly and monthly goals
And the results are that matters, not the effort
I wish you a great week ahead ๐ | 42 006 | 38 | Loading... |
26 โ Message was hidden by channel owner | 14 388 | 0 | Loading... |
27 โ Message was hidden by channel owner | 17 271 | 1 | Loading... |
Challenging programming projects you should try:
๐https://jamesg.blog/2024/02/28/programming-projects/https://jamesg.blog/2024/02/28/programming-projects/
๐ 17
Who's here?
We've asked for a free link to a paid channel, for our subs.
x2-x3 Signals here
๐ CLICK HERE TO JOIN ๐
๐ CLICK HERE TO JOIN ๐
๐ CLICK HERE TO JOIN ๐
โ๏ธJOIN FAST! FIRST 1000 SUBS WILL BE ACCEPTED
๐ 14๐ 8
๐๐ผ๐ ๐๐ผ ๐ฐ๐ผ๐ฑ๐ฒ ๐๐ถ๐๐ต ๐๐ถ๐๐๐๐ฏ ๐๐ผ๐ฝ๐ถ๐น๐ผ๐?
A recent study by GitHub and Microsoft discovered that AI now authors 46% of new code. They also found that overall developer productivity surged by 55%, leading to more efficient coding processes. When we talk about AI-powered coding, we mainly talk about GitHub Copilot.
But ๐ต๐ผ๐ ๐๐ถ๐๐๐๐ฏ ๐๐ผ๐ฝ๐ถ๐น๐ผ๐ ๐๐ผ๐ฟ๐ธ๐?
The process goes in the following steps:
๐ญ. ๐ฆ๐ฒ๐ฐ๐๐ฟ๐ฒ ๐ฝ๐ฟ๐ผ๐บ๐ฝ๐ ๐๐ฟ๐ฎ๐ป๐๐บ๐ถ๐๐๐ถ๐ผ๐ป: Your prompts are securely sent to Copilot, ensuring data privacy.
๐ฎ. ๐๐ผ๐ป๐๐ฒ๐
๐๐๐ฎ๐น ๐๐ป๐ฑ๐ฒ๐ฟ๐๐๐ฎ๐ป๐ฑ๐ถ๐ป๐ด: Copilot analyzes the code around your cursor, the file type, and other open files to offer relevant suggestions.
๐ฏ. ๐๐ผ๐ป๐๐ฒ๐ป๐ ๐ณ๐ถ๐น๐๐ฒ๐ฟ๐ถ๐ป๐ด: It filters out personal data and inappropriate content, focusing solely on generating helpful code.
๐ฐ. ๐๐ผ๐ฑ๐ฒ ๐ด๐ฒ๐ป๐ฒ๐ฟ๐ฎ๐๐ถ๐ผ๐ป: Based on the intent identified in your prompts, Copilot crafts code suggestions that align with your coding style and project standards.
๐ฑ. ๐จ๐๐ฒ๐ฟ ๐ถ๐ป๐๐ฒ๐ฟ๐ฎ๐ฐ๐๐ถ๐ผ๐ป: Here, we can decide whether to use, tweak, or reject Copilot's suggestions.
๐ฒ. ๐๐ฒ๐ฒ๐ฑ๐ฏ๐ฎ๐ฐ๐ธ ๐น๐ผ๐ผ๐ฝ: Copilot learns from your interactions, improving its suggestions. Every time you tweak or reject its ideas, he knows from it. It employs techniques like zero-shot (asking without examples), one-shot (asking with an example), and few-shot learning (providing multiple examples) to adapt to our instructions, whether you provide examples or not.
๐ณ. ๐ฃ๐ฟ๐ผ๐บ๐ฝ๐ ๐ต๐ถ๐๐๐ผ๐ฟ๐ ๐ฟ๐ฒ๐๐ฒ๐ป๐๐ถ๐ผ๐ป: It remembers past prompts and interactions, making future suggestions more accurate.
๐ 28๐ 5
โ Message was hidden by channel owner
๐ 24๐ 2
โ Message was hidden by channel owner
๐ 6๐ 1
So, how they solved it? ๐ง๐ต๐ฒ๐ ๐ณ๐ถ๐ฟ๐๐ ๐๐ฟ๐ถ๐ฒ๐ฑ ๐๐ผ ๐๐ป๐ฑ๐ฒ๐ฟ๐๐๐ฎ๐ป๐ฑ ๐ต๐ผ๐ ๐๐ต๐ฒ ๐๐๐๐๐ฒ๐บ ๐ฝ๐ฒ๐ฟ๐ณ๐ผ๐ฟ๐บ๐. They tracked what Elixir processes were doing, if they were stuck waiting on something, etc. They recorded the event types, how many of each kind of message they received, and their processing times. In addition, they tried to understand how much memory they use, the performances of garbage collectors, etc.
After the analysis, they ๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ๐ฑ ๐๐ต๐ฒ ๐ณ๐ผ๐น๐น๐ผ๐๐ถ๐ป๐ด ๐๐๐ฟ๐ฎ๐๐ฒ๐ด๐:
๐ญ. ๐ฃ๐ฎ๐๐๐ถ๐๐ฒ ๐๐ฒ๐๐๐ถ๐ผ๐ป๐: Discord significantly reduced the amount of data processed and sent by differentiating between active and passive user connections, cutting the fanout work by 90% for large servers.
๐ฎ. ๐ฅ๐ฒ๐น๐ฎ๐๐: Implementing a relay system (read - multithreading) allowed Discord to split the fanout process across multiple machines, enabling a single guild to utilize more resources and support more prominent communities. Relays maintain connections to the sessions instead of the guild and are responsible for doing fanout with permission checks.
๐ฏ. ๐ช๐ผ๐ฟ๐ธ๐ฒ๐ฟ ๐ฝ๐ฟ๐ผ๐ฐ๐ฒ๐๐๐ฒ๐ ๐ฎ๐ป๐ฑ ๐๐ง๐ฆ: To maintain server responsiveness, Discord employed worker processes and Erlang Term Storage (ETS) for operations requiring iteration over large sets of members, thus avoiding bottlenecks in the guild process. ETS is an in-memory database that supports the ability of multiple Elixir processes to access it safely. This enables the creation of a new worker process and passes the ETS table so this process can run expensive operations and offload the central guild server.
๐https://discord.com/blog/maxjourney-pushing-discords-limits-with-a-million-plus-online-users-in-a-single-server
๐ 23
๐๐ผ๐ ๐ฑ๐ผ๐ฒ๐ ๐๐ถ๐๐ฐ๐ผ๐ฟ๐ฑ ๐ต๐ฎ๐ป๐ฑ๐น๐ฒ ๐ฎ ๐บ๐ถ๐น๐น๐ถ๐ผ๐ป ๐ผ๐ป๐น๐ถ๐ป๐ฒ ๐๐๐ฒ๐ฟ๐ ๐ถ๐ป ๐ฎ ๐๐ถ๐ป๐ด๐น๐ฒ ๐๐ฒ๐ฟ๐๐ฒ๐ฟ?
As time passed, the overall size of Discord's user base, including its most prominent communities, has grown massively. This affected servers that started to slow down and hit their throughput limits. So, they needed to scale individual Discord servers from tens of thousands to millions of concurrent users.
Whenever someone sends a message on Discord or joins a channel, they need to update the date UI of everyone online on that server. They call that server a "๐ด๐๐ถ๐น๐ฑ," which runs in a ๐๐ถ๐ป๐ด๐น๐ฒ ๐๐น๐ถ๐
๐ถ๐ฟ ๐ฝ๐ฟ๐ผ๐ฐ๐ฒ๐๐, while there is another process (a "๐๐ฒ๐๐๐ถ๐ผ๐ป") for each connected client. The guild process tracks sessions of users who are members of that guild and are responsible for actions to those sessions. When sessions get updates, forward them to the web socket socket to the client.
The main issue is that ๐ฎ ๐๐ถ๐ป๐ด๐น๐ฒ ๐บ๐ฒ๐๐๐ฎ๐ด๐ฒ ๐ป๐ฒ๐ฒ๐ฑ๐ ๐๐ผ ๐ด๐ผ ๐๐ผ ๐๐ต๐ฒ ๐ป๐๐บ๐ฏ๐ฒ๐ฟ ๐ผ๐ณ ๐ฝ๐ฒ๐ผ๐ฝ๐น๐ฒ ๐ผ๐ป๐น๐ถ๐ป๐ฒ on that server, which means if a server has 1000 people online and they all send a message once, that's 1 million notifications.
๐ 13๐ 1