Every part needs to do function as God has placed the parts in order for a church to be effective in ministry.
Ministry is serving Christ and his church. To say you don't have time to minister is to say you don't have time to serve the bride of Christ. Ministry shouldn't be confused with mission. If I mow my neighbor's lawn I'm doing mission. (unless he is a member of the church, then that would be ministry) If I love my church (that's the people, not the building) I should want to serve my church.
To be a balanced, spiritually healthy, believer a follower of Christ must be actively involved in ministry in their local church. I'm spiritually unhealthy if I'm not giving of my time, talents, personality, resources. How much time in ministry is enough time in ministry? Great question! You tell me. Some of the busiest people I know in their business, work and families are some of the ones who are most involved in ministry also.
Great joy is found in serving where God has SHAPED you to serve. Your church is in poor shape when every part isn't functioning where God has placed him/her.
Where do you serve in your church? Is it a fit? Does is fit your spiritual gift(s), your personality, your experiences? Are you passionate about the ministry you are involved in?