Discover how the 4soft team supports Cryptio in building data ingestion infrastructure.
Tech Details
• Rust
• TypeScript
• Kubernetes
• AWS
Cryptio's primary task is to ingest a distributed, often incomplete transaction history from hundreds of sources (blockchains, crypto exchanges, custodians, and DeFi protocols) and transform it into consistent, fully auditable accounting data. The platform processes information on a large scale, ensuring accounting-grade quality, which allows financial departments to correctly calculate the cost basis, perform account reconciliation, and export data to traditional ERP systems.
Customers publicly using Cryptio include market leaders such as Uniswap Labs, Gnosis, Circle, Gemini, and ConsenSys.
Project Challenge: Rigorous Data Completeness in the Web3 World
Unlike systems focused on anti-money laundering (AML/compliance), accounting systems present engineers with entirely different definitions of success. The main challenge here is the absolute correctness and completeness of financial data.
• Multiple incompatible sources: Various systems (e.g., EVM chains, Cosmos, Solana, exchange ledgers, or custodian interfaces like BitGo and Anchorage) must be mapped to a single, consistent transaction model.
• Zero data loss requirement: In an accounting system, no staking reward, sub-account transfer, or network fee can be lost. A so-called "silent success" of an import, where no transactions were fetched due to errors, is treated as a critical system bug.
• Scale and API limitations: Ingesting massive volumes of historical and current transaction data requires continuous management of rate limits from various providers. Each source imposes different pagination and historical data access rules.
Building end-to-end connectors: We create integrations from scratch with new exchanges and networks (including Bitpanda, GMO, MegaETH, Moca, Scroll, PulseChain). The connectors are delivered comprehensively – from credential definition, through validation, to enabling features in the SaaS dashboard on the user's side.
Advanced custodian support: We implement deep integrations, including for BitGo and Anchorage. This requires advanced business logic, e.g., properly mapping OTC transactions as a single trade rather than independent input/output transfers, which would ruin cost basis calculations. We also handle wallet filtering, fees, and checkpoints.
Expanding data pipelines and resumability: We design parallelized worker-type processes that utilize shared rate limits and persistent checkpoints. This allows multi-hour imports to be easily resumed in case of interruptions. The system also performs subsequent runs for "pending" transactions, capturing their final state.
Migrations to internal indexers (gRPC): The team carries out smooth chain migrations (e.g., Polygon, Avalanche, Arbitrum, Ripple) from external REST-based explorers to Cryptio's proprietary, internal gRPC-based indexers. This ensures better control, more complete data, and bypassing external API limits, all without destroying the history of old imports.
A key operational element is maintaining the integrity of the collected data. The 4soft team has implemented automated control mechanisms that continuously verify data coverage gaps and flag slow or failing imports. Analytics are based on Grafana dashboards, which help quickly identify unusual infrastructure behavior.
Artificial Intelligence (AI) is treated in the project as an equal participant in the software development lifecycle. We use AI-based tools for programming, testing solutions, code review, and assisting in the analysis of operational incidents. We utilize reproducible instructions that give agents appropriate context from the codebase, allowing 4soft engineers to make the final technical decisions and ensure the highest quality of the production environment.
The 4soft team's work directly translates to Cryptio's business effectiveness:
• We have ensured a high throughput in the development and deployment of new integrations (both blockchains, as well as exchanges and custodians) in an end-to-end model.
• We have successfully optimized data quality and fixed completeness gaps in the largest live integrations (e.g., BitGo, Coinbase Prime, Gate.io, Deribit, Bitstamp).
• We have brought deep domain expertise in the area of accounting and financial data quality from the Web3 world, providing auditors with a foundation for reliable crypto settlements.
