What should Muslim men wear around the house with family?

It depends on who is in the room. With your wife there is no limit, while around your mother, sisters, daughters and children you should keep at least the navel-to-knee area covered. Going shirtless at home is not forbidden in itself, but a T-shirt is better manners when women or teenagers are present.

With your wife

A man asked the Prophet ﷺ whom he should hide his private parts from. The Prophet ﷺ answered, "Conceal your private parts except from your wife" (Sunan Abi Dawud 4017). Between spouses there is no restriction, and dressing for each other is part of a loving marriage. Use common sense when children or guests are around.

With your mother, sisters and daughters

These women are mahram, meaning relatives you can never marry, so you do not need the full clothing you would wear outside. The minimum is the navel to the knee, according to most scholars, so your chest, back and arms are not awrah. Technically, being shirtless is allowed. But the Prophet ﷺ taught that modesty is part of faith (Sahih al-Bukhari 9), and it is kind to put on a T-shirt, especially around teenage daughters, sisters and an elderly mother. Boxers alone are not enough, so wear pyjama bottoms or shorts that reach the knee. See Can Muslim men go shirtless?

With in-laws, cousins and guests

Your brother's wife, your wife's sister, cousins, housemates and visitors are not mahram, so dress fully: a shirt and long trousers or shorts to the knee. The Prophet ﷺ warned men about entering on women, and when a man asked about the husband's relatives, he replied that they are death itself (Sahih al-Bukhari 5232). Scholars explain this as a warning to be extra careful with relatives who mix freely at home. Avoid being alone with a non-mahram woman, even in a family house.

Children and privacy

Very young children have no awrah rules, but the Quran teaches older children to respect your privacy at three times of the day, when people are most likely to be undressed:

لِيَسْتَـْٔذِنكُمُ ٱلَّذِينَ مَلَكَتْ أَيْمَـٰنُكُمْ وَٱلَّذِينَ لَمْ يَبْلُغُوا۟ ٱلْحُلُمَ مِنكُمْ ثَلَـٰثَ مَرَّٰتٍ ۚ مِّن قَبْلِ صَلَوٰةِ ٱلْفَجْرِ وَحِينَ تَضَعُونَ ثِيَابَكُم مِّنَ ٱلظَّهِيرَةِ وَمِنۢ بَعْدِ صَلَوٰةِ ٱلْعِشَآءِ ۚ ثَلَـٰثُ عَوْرَٰتٍ لَّكُمْ "Let those bondspeople in your possession and those of you who are still under age ask for your permission to come in at three times: before dawn prayer, when you take off your outer clothes at noon, and after the late evening prayer. These are three times of privacy for you." Quran 24:58

Those times are before Fajr, at the midday rest and after Isha. Teach children to knock and wait, and do the same for them as they grow. See What is the awrah of a child in Islam?

A simple home routine

Keep a T-shirt and knee-length shorts or pyjama trousers for daily use.
Dress fully before leaving your bedroom when you share the house with others.
Knock on bedroom and bathroom doors, and expect the same from others.
If you pray at home, put a shirt on, since shoulders should be covered.

Women have a matching set of questions. See What can I wear at home in front of my father and brothers?

Key points

With your wife, there is no restriction.
With mahram women, cover at least the navel to the knee, and a T-shirt is better.
With non-mahram relatives and guests, dress fully.
Children must knock at the three times of privacy (Quran 24:58).

Tip: Agree a simple house rule: everyone knocks, and everyone is dressed before they come to the kitchen or living room. It keeps the home comfortable for adults and teenagers.