The first time I visited a proper onsen, I spent more time worrying about doing something wrong than I did actually relaxing in the water, which is exactly backwards from the point of the whole thing. A few years and a lot of hot springs later, the rules make sense to me now, and none of them are complicated once somebody actually explains them.
Wash before you soak
This is the one rule that matters more than any other. Onsen water is shared, and it is meant to be clean, restorative water, not bathwater. Every onsen has a washing area before the bath itself, usually a row of low stools with handheld showers and soap. You sit down, you scrub thoroughly, rinse completely, and only then do you step into the communal bath. Skipping this is the fastest way to get a pointed look from other bathers, and honestly, it is the whole logic of the place. The bath is for soaking, not for washing.
Yes, it is usually swimsuit-free
Traditional onsen are bathed in without clothing, which trips up a lot of first-time visitors from swimsuit-culture countries. It is not sexualized the way it might feel from the outside, it is simply the norm, and everyone else in the room is doing the same thing and paying you no attention at all. Baths are separated by gender in almost all traditional facilities. If the idea of communal nude bathing is a hard no for you, look for a ryokan or hotel with a private bath option, often called a kashikiri-buro, which you can reserve for yourself or your travel companions.
The small towel is not for the bath
You will be given, or can rent, a small modesty towel. Use it to cover yourself as you walk around the bathing area, and do not let it touch the bathing water itself, most people fold it and set it on their head or on the edge of the bath while soaking. It feels like a small detail, but it is one of those unwritten rules that locals notice.
The tattoo question
This is the one that surprises the most visitors. Tattoos have historically been associated with the yakuza in Japan, an association reinforced for decades by films and popular culture, and as a result many onsen still ban visible tattoos outright, with signs posted at the entrance. The good news is this is slowly changing, especially in areas that see a lot of international tourism, and a growing number of onsen towns now list themselves as tattoo-friendly. If you have tattoos and want a guaranteed stress-free soak, your best options are seeking out one of these explicitly tattoo-friendly facilities, covering smaller pieces with a waterproof tattoo sticker, or booking a private bath, which sidesteps the issue completely and is honestly a lovely experience in its own right.
Keep the volume down
Onsen culture leans quiet. Conversation happens, but at a low murmur, and phones generally do not come out at all near the water, both as a courtesy to other bathers and because most onsen do not allow cameras in the bathing area for obvious privacy reasons. Long hair gets tied up so it does not touch the water. Swimming and splashing are not really a thing here, the point is to sit still and let the heat do its work.
Pace yourself
Onsen water often runs hotter than what you are used to, sometimes 40 to 44 degrees Celsius, and the minerals in it can be more intense on your system than a regular bath. Ease in slowly, take breaks sitting on the edge if you feel lightheaded, and drink water before and after. I learned this one the hard way after standing up too fast and needing to sit down again immediately.
Why it is worth the small learning curve
Once you get past the first-timer nerves, an onsen is one of the most genuinely restorative things Japan does well. There is a particular kind of quiet that settles over a bath at dusk, steam rising off the water, mountains or a garden just visible past the edge, nobody talking much, that I have not really found anywhere else. The etiquette exists to protect that quiet, not to make visitors feel unwelcome. Follow it, and you will fit right in.