Is a house worth it?

I have continued to claim at times around my family that owning our current house is actually a bad investment that prevents us from achieving financial independence.   

They point out that it has been a good investment, since it will double in value at about the 12th year that we owned it.   So we received $200K equity.

However, they don't take into account that due to having a larger house, we've had large expenses, and larger insurance costs, and heating and cooling costs.   So let's say that is about $200K more in expenses than a little house would have been, cancelling out the equity.   

Meanwhile, little houses are doing amazing in the housing market, and are in high demand.  Our little house would have about $150K equity right now.  So the little house is actually a better investment.  It would be paid off for several years now and all of our income could be going to retirement or college savings, or whatever we need.  Instead, we are targeting to have our current house paid off in about 6 years, and are behind on college savings.

That being said, we would have been driven crazy in the little house we had before. We have a lot of family members with high energy and emotion, and there would have been nowhere to go to escape.  So I think it's been an ok for our family to sacrifice the financial independence for that.  I just wish the house we had bought was not such a money pit of renovations.  But maybe every house is like that.

If I had it my way, I would have minimized the cost of dwelling, and focused more financial energy on savings, and using spending for outside experiences, hobbies, quality vehicles, etc. I feel the house has been too heavy of an expense.  But everyone disagrees with me, so I think the goal is just to accept our previous decisions and to be content.