About MetalNet

MetalNet is a blazing fast, highly efficient, header-only custom Convolutional Neural Network (CNN) library implemented in pure C++20. It aims to provide raw CPU performance capable of matching or exceeding frameworks over optimized CPU regimes, utilizing native vector expressions and robust threading models. The entire library requires zero external dependencies beyond a C++20 compiler, OpenMP, and AVX2/FMA support.

Architecture Overview

The core philosophy of MetalNet surrounds the concepts of Zero-Overhead abstractions and Native Hardware Mapping. Everything from the Directed Acyclic Graph (DAG) construction to deep layer fusions is engineered to exploit modern CPU caches effectively.

• No External Dependencies: Standalone, leveraging modern C++ standard templates alongside OpenMP and AVX2 intrinsics. 📘 C++20 Reference
• Fluent Graph Builder: Offers an incredibly fluent sequence builder allowing code like model << conv2d(1, 64, 3) << relu();
• Flexible Extensibility: A polymorphic Layer interface making it trivially easy to add new operations or mathematical transformations while remaining plugged into the automatic differentiation mechanics.

How to Use

Using MetalNet mirrors high-level structures from Keras or PyTorch while retaining C++ execution profiles. The fluent API allows building complex architectures in a few lines.

Building a Model Example

#include "MetalNet/MetalNet.h"

using namespace MetalNet;

Model build_vgg_model() {
    Model model;
    // Fluent API: layers are chained with operator<<
    model << conv2d(1, 64, 3, 1, 1) << batchnorm2d(64) << leaky_relu(0.01f)
          << conv2d(64, 64, 3, 1, 1) << batchnorm2d(64) << leaky_relu(0.01f)
          << maxpool2d(2, 2) 
          << flatten() << dense(128 * 7 * 7, 10);
          
    // Compile precomputes memory layout and DAG
    model.compile({128, 1, 28, 28}); // batch, channels, height, width
    return model;
}

Inference Example

Model inf_model = build_vgg_model();
inf_model.eval(); // drops gradient buffers, minimizes memory footprint

Tensor single_img(1, 1, 28, 28);
// ... load image data ...

Tensor prediction = inf_model.forward(single_img);
// prediction contains logits or probabilities

Additional Example (Residual Block): Because MetalNet uses a DAG, you can manually add skip connections by directly connecting layer outputs. Refer to the repository examples/ for advanced topologies.

How to Compile and Run

Because MetalNet leverages highly specific CPU registers and threading paths, the compilation flags must strictly enforce optimization ceilings.

git clone https://github.com/KunwarPrabhat/CustomCNN
cd CustomCNN

Compilation Syntax (Detailed)

To compile the project with GCC/G++, invoke the following instruction chain from your terminal. It explicitly maps AVX2 layout pipelines, enables OpenMP scheduling arrays, enforces strict Standard C++20 compliance, and removes unmathematical constraints via fast-math.

g++ -std=c++20 -O3 -mavx2 -mfma -fopenmp -march=native -ffast-math -I./include main.cpp -o benchmark.exe -lpsapi

Flag breakdown: -mavx2 -mfma enable AVX2 & FMA instructions; -fopenmp parallelizes tensor ops; -march=native optimizes for host CPU; -ffast-math enables aggressive floating-point optimizations (FTZ/DAZ). -lpsapi (Windows) measures memory; omit on Linux.

Run the executable with ./benchmark.exe.

Optimization Techniques Used

Implementation of Directed Acyclic Graph (DAG)

Gradients propagate through an internal DAG layout mapped dynamically via Layer* output_nodes and input_nodes lists. This design safely models non-sequential topologies like Residual Connections natively without tracking external backprop indices. Compilation steps traverse the DAG hierarchically to pre-allocate exact graph tensors.

Example: A residual block y = F(x) + x is represented as two separate paths merged by an Add layer. MetalNet's DAG executor automatically handles gradient branching. 📘 DAG Theory

Zero Allocation Techniques

The library features a strictly Zero-Allocation Hot-Path. The compile() method creates thread-local gradient buffers and dynamically sizes the graph prior to loop execution. Further, a custom C++ std::allocator (NoInitAllocator) bypasses the OS 200ms default zero-initialization trap of std::vector::resize(), executing tensor allocation under ~1ms. When moving to inference via model.eval(), MetalNet actively triggers memory shrinkage on gradients down to a completely flat footprint.

// NoInitAllocator prevents zero-fill
template<typename T>
struct NoInitAllocator : public std::allocator<T> {
    void construct(T*) noexcept { /* no-op */ }
};

C++ Allocator Docs | std::vector::resize performance

Vectorization (AVX2 & FMA)

A major throughput breakthrough came via completely circumventing C++ array iterations. _mm256_fmadd_ps and _mm256_loadu_ps unaligned intrinsic expressions manually instruct the SIMD processor lines. MetalNet processes 8 floats per cycle natively. Fused Multiply-Add (FMA) mathematically folds arithmetic into singular pipeline pulses without precision degradation.

__m256 vec = _mm256_loadu_ps(&data[i]);
__m256 res = _mm256_fmadd_ps(vec, weight, accum);

Intel AVX2+FMA Intrinsics Guide

Advanced Multithreading (OpenMP)

Loop-collapsed threading arrays orchestrate scaling via #pragma omp parallel for schedule(static). MetalNet actively dodges the "OpenMP Threading Tax" by gating OpenMP directives entirely for latency-sensitive workloads like single-sample inference (batch size 1). When processing batches, OpenMP spreads independent image patches natively over Logical Threads.

Thread affinity: Set OMP_PROC_BIND=true and OMP_PLACES=cores to reduce context switching. OpenMP Thread Affinity

Efficient Matrix Multiplication (GEMM)

MetalNet pivoted heavily from slow, naïve nested 6D direct convolutions. To map to modern hardware effectively, Im2Col (Image-to-Column) algorithms are used alongside highly refined Block GEMM dense structures to transform the Convolution routine into heavily parallelized matrix multiplications. To avoid memory bloats to 10GB+, subsequent paths utilize Implicit GEMM paradigms that load patch memory dynamically to subvert memory duplication.

📄 Implicit GEMM for CNNs (paper) | BLAS GEMM standard

L1/L2 Cache Optimizations

Register Accumulation ensures sums are computed internally via strict __m256 accumulators and only stored back to RAM memory exactly once. This removes millions of Read-After-Write (RAW) cache hazards occurring in the L1 pipeline. Standard arrays are loop-tiled in sizes matching L2 cache envelopes (BLOCK_SIZE = 64) maximizing data permanence per thread computation.

Loop Tiling (Cache Blocking) | Intel Optimization Manual

Problems Faced & Bottlenecks

Architectural & Implementation Bottlenecks

Memory Management Challenges

Mathematical & Convergence Issues

System & Hardware Bottlenecks

Benchmarks vs PyTorch

The comprehensive benchmarks below demonstrate MetalNet matching and often drastically exceeding the CPU efficiencies of standard baseline implementations, maintaining strict sub-400MB footprints. All tests run on Intel Xeon/AMD Ryzen with 8 physical cores, AVX2 enabled.

MetalNet (Standard)

PyTorch (Standard)

MetalNet (MNIST Performance)

PyTorch (MNIST Performance)

MetalNet (CIFAR Performance)

PyTorch (CIFAR Performance)

MetalNet (2X Deep CNN)

PyTorch (2X Deep CNN)

MetalNet (4X Extreme Wide CNN)

PyTorch (4X Extreme Wide CNN)