Leadership in the church is not based upon power, but upon authority. Therefore being a pastor/elder is not a position of power, but a position of authority. However, it is not an authority inherent in the person, it is a delegated authority because of the position held. It is delegated from God to the elder(s), and also delegated from the congregation to the elder(s).
It is God who has created the position of pastor/overseer/elder and laid out the qualifications for the office, and it is the recognition by the congregation of those who meet God's qualifications (I Timothy 3, Titus 1) that bring the man of God to that position. So God calls the pastor/elder to that office from the congregation with their approval.
The pastor/elder is to lead, not lord it over the people (Hebrews 13:7, I Peter 5:2-3); and the people are to voluntarily submit to his leadership and follow his example (Hebrews 13:7, 17; I Corinthians 16:16), just as a wife is to voluntarily submit to her husband.
As such, the man of God is accountable to both God and man, as both were instrumental in placing him in that office; and both delegated to him the authority which the office holds.