In this paper, a flexible block-wise image compression architecture with restoration network is proposed to improve the performance of learned image compression. In contrast to existing learned compression algorithms which perform the imaged based transform coding, our architecture splits the input image into non-overlapped blocks and compresses each block adaptively based on their local contents. In order to reduce the artifacts caused by the block partition, block fusion network is subsequently introduced to overcome the drawbacks of block-wise independent coding. Further, quantization fine-tuning process is utilized to reduce the difference of quantization operations between the training process and testing process. Experimental results show that the proposed network architecture has the ability to obtain significant compression gains compared with common compression standards including Versatile Video Coding (VVC) and achieves comparable performance with state-of-the-art learned image codec using less parameters.