We propose a heterogeneous architecture coupling 8 RISC-V cores with an IMA in a shared-memory cluster, analyzing the benefits and trade-offs of in-memory computing in a MobileNetV2 bottleneck. We show that while pointwise layers achieve significant speed-ups over software, on depthwise layers inefficient mapping on the accelerator leads to a significant area/throughput trade-off. Our solution where pointwise convolutions are executed on IMA while depthwise on the cores achieves a speed-up of 3x over SW while saving 50% of area when compared to an all-in IMA solution with similar performance.