FSM_ENCODING controls encoding on the state machine. Typically, the Vivado tools choose an encoding protocol for state machines based on heuristics that do the best for the most designs. Certain design types work better with a specific encoding protocol.
FSM_ENCODING can be placed on the statemachine registers. The legal values for this are "one_hot", "sequential","johnson","gray","user_encoding" and "none". The "auto" value is the default, and allows the tool to determine best encoding. The "user_encoding" value tells the tool to still infer a statemachine, but to use the encoding given in the RTL by the user.
The FSM_ENCODING attribute can be set in the RTL or the XDC.