Managing contributions from multiple developers with diverse specializations has been a key aspect of the TechDico project. Initially, the project was organized across multiple repositories, each focusing on different components and features. However, as the project evolved and the need for tighter integration and streamlined collaboration became apparent, we transitioned to a single monorepo. This shift allowed us to centralize our codebase, simplifying dependency management and enabling more cohesive development practices. The monorepo structure has facilitated better coordination among team members, reduced merge conflicts, and improved the overall efficiency of our development workflow.
To ensure smooth collaboration and continuous improvement, I introduced a continuous flow methodology inspired by Kanban. This approach avoids time-boxed sprints, allowing improvements to be delivered continuously. Tasks are prioritized and assigned with careful attention to minimize merge conflicts and limit the number of concurrent tasks in progress at any time. This strategy helps maintain a steady workflow, reduces bottlenecks, and ensures that the team can focus on delivering high-quality features efficiently. The entire team operates fully remotely, leveraging digital tools to facilitate communication and project management.
Innovation is at the heart of TechDico’s development strategy. We strive to achieve minimal latency in every operation, ensuring that users experience swift and responsive interactions with the platform. Handling terabytes of data efficiently is a cornerstone of our approach, leveraging advanced filtering and categorization techniques to manage this vast amount of information effectively. By leveraging AI, particularly BERT classification, we can process and classify data with unprecedented accuracy, enhancing the relevance and quality of the information we deliver. The use of leading-edge hardware and software technologies ensures continuous improvements in our capabilities and performance. Additionally, sophisticated caching mechanisms further reduce latency and optimize data retrieval, ensuring that users always have access to the most current and relevant information.
The use of multiple technology stacks in TechDico is driven by the need for constant evolution, interoperability, and parallelization. As the project has grown since 2017, integrating new features and improving performance has required adopting modern technologies. Initially, the platform was designed to be compatible with IE8, but it has since transitioned to more advanced frameworks like ReactJS and NodeJS, which offer better user interfaces and server-side efficiency.
Different components of TechDico necessitate specific tools to function optimally. Golang is utilized extensively for its efficiency in handling concurrent processes, including data retrieval, preparation, filtration, and formatting for display by ReactJS. Additionally, Golang plays a crucial role in normalizing, filtering, classifying, and ingesting large volumes of data. Elasticsearch is employed for its powerful search capabilities, and MariaDB ensures reliable data storage. This combination allows seamless communication between various parts of the system, enhancing overall functionality and enabling the platform to meet diverse user needs.
TechDico initially started on AWS, but the high costs, high bandwidth requirements, and predictable load patterns prompted a move to dedicated servers for Elasticsearch and later for the API and FrontEnd SSR. This transition was part of a constant evolution towards more cost-effective solutions, adopting a hybrid cloud and dedicated server approach for data preparation, filtering, and ingestion.
Recently, we began migrating some services, such as Elasticsearch, from x86 to ARM architecture due to the better price-performance ratio. Our deployment process is streamlined with GitLab CI and Ansible, ensuring efficient and reliable updates. We utilize a multi-stage pipeline that includes building everything into Docker containers, testing each component individually, and deploying using systemd, Kubernetes, or static assets as needed.
In cases where a cache wipe is necessary, we perform a clean-up of RocksDB and invalidate the CDN cache via the Cloudflare API. This ensures that our users always have access to the most up-to-date information.
We utilize K3S on bare metal for SSR servers that require more parallelization and are in the process of moving other components to Kubernetes. This shift to Kubernetes is driven by several factors:
To enhance security and manage traffic efficiently, we implemented a MeshVPN to interconnect all servers, ensuring secure and seamless communication. We adopted a zero-trust access model to minimize security risks and employed traffic flow management to optimize network performance. This entire setup is automated and monitored using Prometheus, which provides real-time insights and alerts, ensuring the system’s reliability and performance. The setup of a new dedicated, cloud, or K3S server takes only a few minutes, with everything automated using Ansible, further streamlining our operations.
By leveraging these technologies and methodologies, TechDico not only achieves operational efficiency but also reduces costs associated with infrastructure and maintenance. The use of open-source tools and scalable solutions helps in optimizing resource utilization and minimizing overhead expenses.
