Swimming nude with sister?

Last weekend my brother and I were hanging out reminiscing about the past when the topic about us skinny dipping together when we were in elementary school, harmless kid stuff as we saw it. Well we are both adults now and While we were talking it kind of turned in to "I'll do it if you do it" kind of thing and next thing I know we are in the pool with nothing more than just our birthday suits. Nothing sexual happened we were just happily swimming in the pool. Was this wrong since we are brother and sister?