James finds evidence of a problem in the messianic community, especially among the teachers, and he finds the source of the problem to be the yearning, desiring, and craving for power. He must tell them so (James 4:1-3). Next he connects the self-originating issues in a connection with the world (4:4-6):

You adulterous
people, don’t you know that friendship with the world is hatred toward
God? Anyone who chooses to be a friend of the world becomes an enemy of
God. Or do you think Scripture says without reason that the spirit he caused to live in us envies intensely? But he gives us more grace. That is why Scripture says:
   “God opposes the proud
      but gives grace to the humble.”

And the last little paragraph fills up volumes of debate, and we’ll get to that debate in due course.

Like his brother, he posits two options and draws a big line in the sand and says, “Take your pick.” Choose “friendship” with God or choose friendship with the world. James sets up this contrast with “you adulterous people,” which strikes readers today as harsh, esp for someone who has made it clear that use of the tongue is so important. In James’ mind, this is not inconsistent so we should give him a break.

The point is clear: those who are in friendship with the world, seen in misuse of the tongue and lack of wisdom and the acting out of craving desires, hate God. (Friendship and hate, as contrasts, come perhaps from Jesus: Luke 14:26-27.)

This is one serious contrast, and James’ intent is to pry the teachers loose from their self-originating desires. Adultery makes sense here: friendship with God cannot be shared with the world — it’s adulterous in one’s relationship with God.