| This paper considers an Energy Harvesting Wireless Sensor Network (EH-WSN) where nodes have a dual alternative battery system. We aim to design a routing protocol that transmits a given amount of data to a sink using the minimum number of time slots. To this end, we first formulate the routing problem as a Mixed Integer Linear Program (MILP) and model the charging and discharging process at each node. Moreover, we also propose a distributed Multi-Armed Bandit based Routing Algorithm (MABRA), where each node learns the best next hop(s) to forward its data based on the battery and data information of its neighbors. We study how the number of nodes that generate data or sources, solar panel area, and path exploration probability impacts the end-to-end delays computed by MILP and MABRA. Numerical results show that the end-to-end delays computed by MILP increase with the number of sources. It, however, decreases if nodes use larger solar panels. Critically, nodes experience end-to-end delays that are $55\%$ lower than the average end-to-end delay of a competing routing algorithm. |