This paper presents an algorithm-hardware co-optimization method for designing an efficient accelerator architecture for deep neural networks for image signal processing. The proposed method newly defines three metrics to evaluate the efficiency of systolic accelerators. Then the algorithm-hardware co-optimization strategy is applied to find the most promising configurations on both the hardware architecture and the model parameters. Case studies on recent ML-based ISP models show that the proposed method provides the cost-efficient array dimensions compared to the straightforward method and offers a way to adjust the original network to further optimize ISP operations\' efficiency under the given accelerator architecture.